ConcurrentRebalancersTest.TwoConcurrentRebalancers/0: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240503 02:21:10.476238 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2199
I20240503 02:21:10.483883  2321 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:10.851588 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2199
W20240503 02:21:10.930400  4129 connection.cc:535] server connection from 127.20.218.65:34381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930414  4395 connection.cc:535] server connection from 127.20.218.65:43879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930509  2478 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930666  2478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 26 similar messages]
W20240503 02:21:10.930968  4263 connection.cc:535] server connection from 127.20.218.65:42755 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:10.931151 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2330
W20240503 02:21:10.931264  2348 connection.cc:535] server connection from 127.20.218.65:33571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931377  2606 connection.cc:535] server connection from 127.20.218.65:58531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931449  2348 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931547  2348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 16 similar messages]
W20240503 02:21:10.931592  2609 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931669  2609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 31 similar messages]
W20240503 02:21:10.932622  2124 connection.cc:535] server connection from 127.20.218.65:35201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.945475  2478 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946141  2348 consensus_peers.cc:485] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946278  2348 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946297  2609 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946388  2348 consensus_peers.cc:485] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:10.952958  2451 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:11.326215 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2330
W20240503 02:21:11.385838  4132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.386596  4129 connection.cc:535] server connection from 127.20.218.66:32795 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:11.387169  2607 connection.cc:535] server connection from 127.20.218.66:53211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:11.387777 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2460
W20240503 02:21:11.387827  4132 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.388332  4129 consensus_peers.cc:485] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.398114  2577 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:21:11.415127  2606 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.428690  4260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:21:11.429900  4260 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.779556 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2460
I20240503 02:21:11.831924 19041 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.832151 19041 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.833019 19041 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:11.833812  4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "408af0ab5520465e942cc714bfc6c73b" is_pre_election: true
I20240503 02:21:11.833817  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:11.834766  2608 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e9fdaf326bb64edabc43b366df870f83; no voters: 1d36807868fe4a82bca42c25831ce8a2, 408af0ab5520465e942cc714bfc6c73b
I20240503 02:21:11.835180 19041 raft_consensus.cc:2726] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:11.835716  2609 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.858322  4262 connection.cc:535] server connection from 127.20.218.67:56343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:11.858935 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2591
W20240503 02:21:11.859361  2607 connection.cc:535] client connection to 127.20.218.67:38127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:11.860270  4261 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.860800  2607 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.860975  2607 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.861092  2607 consensus_peers.cc:485] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.861204  2607 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.865859  2712 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:11.916859 19310 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.917119 19310 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.918946 19310 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:11.919937  4397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.922715  4397 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:11.926512  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:11.927214  4396 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:21:11.927567 19310 raft_consensus.cc:2726] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:11.982729 18732 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.982975 18732 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.983757 18732 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:11.984827  4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "408af0ab5520465e942cc714bfc6c73b" is_pre_election: true
W20240503 02:21:11.985209  4132 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:11.985525  4132 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2; no voters: 08055479ea104aa290d19955d3720850, 408af0ab5520465e942cc714bfc6c73b
I20240503 02:21:11.985805 18732 raft_consensus.cc:2726] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.239439 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2591
I20240503 02:21:12.244143 16912 raft_consensus.cc:482] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.244369 16912 raft_consensus.cc:504] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:12.245642 16912 leader_election.cc:288] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.248453  4260 leader_election.cc:334] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:12.252852  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "9e699d9f984b4cf2b8b468322759b78c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:12.253546  4262 leader_election.cc:302] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.253839 16912 raft_consensus.cc:2726] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:12.301470  4263 connection.cc:535] server connection from 127.20.218.68:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:12.302199 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4114
W20240503 02:21:12.302376 19312 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:21:11.919131 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.218.68:38103 (local address 127.20.218.71:33021)
0503 02:21:11.919776 (+   645us) negotiation.cc:107] Waiting for socket to connect
0503 02:21:11.920655 (+   879us) client_negotiation.cc:174] Beginning negotiation
0503 02:21:11.920757 (+   102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:21:12.301665 (+380908us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":475,"threads_started":1}
W20240503 02:21:12.302896  4131 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Couldn't send request to peer e9fdaf326bb64edabc43b366df870f83. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:12.303803  4396 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:12.308266  4232 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:12.355073 16912 raft_consensus.cc:482] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.355185 19310 raft_consensus.cc:482] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.355333 19310 raft_consensus.cc:504] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:12.355309 16912 raft_consensus.cc:504] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:12.356127 16912 leader_election.cc:288] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:12.356158 19310 leader_election.cc:288] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:12.356304 19310 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.356451 19310 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.357074 19310 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.358361  4395 leader_election.cc:334] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.358772  4260 leader_election.cc:334] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.358889  4263 leader_election.cc:334] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:12.358969  4397 leader_election.cc:334] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.359007  4263 leader_election.cc:302] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.359092  4397 leader_election.cc:302] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:12.359285 16912 raft_consensus.cc:2726] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.359329 19310 raft_consensus.cc:2726] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:12.359575  4397 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.406826 16912 raft_consensus.cc:482] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.407052 16912 raft_consensus.cc:504] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:21:12.407786 16912 leader_election.cc:288] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.409189  4260 leader_election.cc:334] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.409348  4263 leader_election.cc:334] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.409582  4263 leader_election.cc:302] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.409893 16912 raft_consensus.cc:2726] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.410866 19310 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.411054 19310 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.412015 19310 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.413246  4394 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.413753  4396 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:12.413941  4396 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:12.414227 19310 raft_consensus.cc:2726] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.419021  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:12.470412 19310 raft_consensus.cc:482] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:12.470688 19310 raft_consensus.cc:504] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.471503 19310 leader_election.cc:288] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.472844  4395 leader_election.cc:334] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.472997  4397 leader_election.cc:334] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.473119  4397 leader_election.cc:302] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:12.473393 19310 raft_consensus.cc:2726] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.579079 19310 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:12.579313 19310 raft_consensus.cc:504] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.580107 19310 leader_election.cc:288] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:21:12.581068  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
W20240503 02:21:12.581676  4395 leader_election.cc:334] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
I20240503 02:21:12.581818  4395 leader_election.cc:302] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:21:12.582108 19310 raft_consensus.cc:2726] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:12.653432  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:12.654162 18732 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.654330 18732 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.654474  4194 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408af0ab5520465e942cc714bfc6c73b in term 1.
I20240503 02:21:12.655009 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4114
I20240503 02:21:12.656944  4396 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2, 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850
I20240503 02:21:12.657279 19310 raft_consensus.cc:2781] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:21:12.657444 19310 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.657537 19310 raft_consensus.cc:3037] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:12.659847 19310 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.660475 19310 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.661865  4397 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:12.712978  4396 connection.cc:535] client connection to 127.20.218.69:39741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:12.713310 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4245
W20240503 02:21:12.715031  4396 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Network error: Client connection negotiation failed: client connection to 127.20.218.69:39741: connect: Connection refused (error 111)
I20240503 02:21:12.715152  4396 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:21:12.715456 19310 raft_consensus.cc:2726] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.719637  4362 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:12.824241 19310 raft_consensus.cc:482] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.824481 19310 raft_consensus.cc:504] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:12.824838 19322 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.825004 19322 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.825305 19310 leader_election.cc:288] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:21:12.825824 19322 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.827643  4395 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.828003  4394 leader_election.cc:334] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.828121  4394 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:12.828222  4394 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
W20240503 02:21:12.828410  4396 leader_election.cc:334] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Network error: Client connection negotiation failed: client connection to 127.20.218.69:39741: connect: Connection refused (error 111)
I20240503 02:21:12.828521  4396 leader_election.cc:302] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.828644 19322 raft_consensus.cc:2726] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:12.828753 19310 raft_consensus.cc:2726] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.828994 19310 raft_consensus.cc:482] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.829157 19310 raft_consensus.cc:504] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.829726 19310 leader_election.cc:288] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.831068  4394 leader_election.cc:334] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.831239  4397 leader_election.cc:334] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.831357  4397 leader_election.cc:302] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.831614 19310 raft_consensus.cc:2726] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.927016 19310 raft_consensus.cc:482] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e9fdaf326bb64edabc43b366df870f83)
I20240503 02:21:12.927244 19310 raft_consensus.cc:504] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.927958 19310 leader_election.cc:288] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.929282  4396 leader_election.cc:334] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:13.077709 16912 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:13.078670  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:13.078795  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:13.078934  4326 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408af0ab5520465e942cc714bfc6c73b in term 2.
I20240503 02:21:13.079094 16912 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:13.079701  4394 leader_election.cc:302] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 9e699d9f984b4cf2b8b468322759b78c, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:13.080456 19310 raft_consensus.cc:2726] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:13.080780 16912 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:13.080915 16912 raft_consensus.cc:482] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:13.081045 16912 raft_consensus.cc:504] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:13.081691 16912 leader_election.cc:288] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:13.081836 16912 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:13.081984 16912 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:13.083647 16912 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
W20240503 02:21:13.084133  4261 leader_election.cc:334] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:13.085044  4260 leader_election.cc:334] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:13.085249  4260 leader_election.cc:302] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
W20240503 02:21:13.085795  4261 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:13.085842  4260 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:13.086058  4260 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:13.086194  4260 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:13.086490 19330 raft_consensus.cc:2726] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:13.086632 16912 raft_consensus.cc:2726] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:21:13.089545  4262 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:13.089736  4262 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:13.090090 16912 raft_consensus.cc:2726] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:13.090063 19328 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:13.090596 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4245
W20240503 02:21:13.150774 19332 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:21:13.091666 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.218.70:40555 (local address 127.20.218.71:38285)
0503 02:21:13.092160 (+   494us) server_negotiation.cc:206] Beginning negotiation
0503 02:21:13.092169 (+     9us) server_negotiation.cc:399] Waiting for connection header
0503 02:21:13.150487 (+ 58318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.218.70:40555: BlockingRecv error: recv got EOF from 127.20.218.70:40555 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20240503 02:21:13.150828 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4377
I20240503 02:21:13.156821  4497 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:21:13.392470  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51dbf0fd54b14c2fa21d3e62104c9b87: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:13.485200 19310 raft_consensus.cc:482] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fdaf326bb64edabc43b366df870f83)
I20240503 02:21:13.485471 19310 raft_consensus.cc:504] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:13.486428 19310 leader_election.cc:288] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:13.487200 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4377
I20240503 02:21:13.544466 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2109
I20240503 02:21:13.550570  2168 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:13.899665 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2109
2024-05-03T02:21:13Z chronyd exiting
I20240503 02:21:13.969877 21353 test_util.cc:175] -----------------------------------------------
I20240503 02:21:13.970005 21353 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0

Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN      ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:23.408216 21353 test_util.cc:268] Using random seed: 1014989606
I20240503 02:11:23.420661 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:23.420753 21353 ts_itest-base.cc:113] --------------
I20240503 02:11:23.420795 21353 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:11:23.420845 21353 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:23.420888 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:23Z Disabled control of system clock
I20240503 02:11:23.446707 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:41289
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:41289
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:23.790275 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:23.790629 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:23.790768 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:23.798857 21365 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:23.799075 21365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:23.799156 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:23.799263 21365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:23.799518 21365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:23.809732 21365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:41289
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:23.811199 21365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:23.813454 21365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:23.826650 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.826747 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.826757 21365 server_base.cc:1034] running on GCE node
W20240503 02:11:23.827004 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.828157 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:23.829922 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:23.831192 21365 hybrid_clock.cc:648] HybridClock initialized: now 1714702283831115 us; error 71 us; skew 500 ppm
I20240503 02:11:23.831624 21365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.840116 21365 webserver.cc:460] Webserver started at http://127.20.218.126:35723/ using document root <none> and password file <none>
I20240503 02:11:23.840999 21365 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.841132 21365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.841532 21365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.844527 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3aab06aa055a41cdb6870af66b723634"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-81lq"
I20240503 02:11:23.845489 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3aab06aa055a41cdb6870af66b723634"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-81lq"
I20240503 02:11:23.851495 21365 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:11:23.855491 21379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:23.857089 21365 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:23.857296 21365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3aab06aa055a41cdb6870af66b723634"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-81lq"
I20240503 02:11:23.857561 21365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.877923 21365 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:23.878937 21365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:23.879256 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.900859 21365 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:41289
I20240503 02:11:23.900880 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:41289 every 8 connection(s)
I20240503 02:11:23.902446 21365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:23.905616 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21365
I20240503 02:11:23.905861 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:23.906692 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:23.917174 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634: Bootstrap starting.
I20240503 02:11:23.920433 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.921602 21431 log.cc:824] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:23.925359 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634: No bootstrap required, opened a new log
I20240503 02:11:23.933563 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } }
I20240503 02:11:23.934062 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.934183 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aab06aa055a41cdb6870af66b723634, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.934975 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } }
I20240503 02:11:23.935281 21431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:23.935421 21431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:23.935595 21431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.938223 21431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } }
I20240503 02:11:23.938887 21431 leader_election.cc:302] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aab06aa055a41cdb6870af66b723634; no voters: 
I20240503 02:11:23.940317 21431 leader_election.cc:288] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:11:23.940474 21436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:23.943774 21436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [term 1 LEADER]: Becoming Leader. State: Replica: 3aab06aa055a41cdb6870af66b723634, State: Running, Role: LEADER
I20240503 02:11:23.944445 21436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } }
I20240503 02:11:23.944819 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.947520 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aab06aa055a41cdb6870af66b723634. Latest consensus state: current_term: 1 leader_uuid: "3aab06aa055a41cdb6870af66b723634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } } }
I20240503 02:11:23.947888 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.947820 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aab06aa055a41cdb6870af66b723634" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aab06aa055a41cdb6870af66b723634" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 41289 } } }
I20240503 02:11:23.948096 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.950037 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:23.955108 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:23.965394 21444 catalog_manager.cc:1347] Generated new cluster ID: 43a89def7fde401ba140a6b728ec5fae
I20240503 02:11:23.965539 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:23.988520 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:23.990505 21444 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:24.002466 21444 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3aab06aa055a41cdb6870af66b723634: Generated new TSK 0
I20240503 02:11:24.003438 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:24.021286 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:11:24.400213 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.400561 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.400709 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.408618 21455 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:24.408769 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.408937 21455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:11:24.418557 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.420014 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.421934 21455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.435042 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.435042 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.435109 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.435626 21455 server_base.cc:1034] running on GCE node
I20240503 02:11:24.437258 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.438668 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.439913 21455 hybrid_clock.cc:648] HybridClock initialized: now 1714702284439860 us; error 48 us; skew 500 ppm
I20240503 02:11:24.440289 21455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.443010 21455 webserver.cc:460] Webserver started at http://127.20.218.65:40195/ using document root <none> and password file <none>
I20240503 02:11:24.443862 21455 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.443995 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.444348 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.447449 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.448377 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.454784 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240503 02:11:24.459178 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:24.460748 21455 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:11:24.460959 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.461189 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.498399 21455 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:24.499331 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.499617 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.500643 21455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.502333 21455 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.502437 21455 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:24.502552 21455 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.502621 21455 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:24.537971 21455 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:39543
I20240503 02:11:24.538022 21581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:39543 every 8 connection(s)
I20240503 02:11:24.539369 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:24.545692 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21455
I20240503 02:11:24.545914 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:24.549445 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:24.553542 21582 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:24.553897 21582 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.554800 21582 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:24.557132 21396 ts_manager.cc:194] Registered new tserver with Master: 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:24.558779 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:59487
W20240503 02:11:24.932361 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.932718 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.932868 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.941671 21586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:24.941818 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.942013 21586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:11:24.953409 21586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.954885 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.957599 21586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.971378 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.971349 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.971419 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.972021 21586 server_base.cc:1034] running on GCE node
I20240503 02:11:24.972704 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.973846 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.975082 21586 hybrid_clock.cc:648] HybridClock initialized: now 1714702284975030 us; error 56 us; skew 500 ppm
I20240503 02:11:24.975455 21586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.978680 21586 webserver.cc:460] Webserver started at http://127.20.218.66:40429/ using document root <none> and password file <none>
I20240503 02:11:24.979497 21586 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.979626 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.979952 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.982868 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "efb50bbbedb9457f88533cc7e32c793a"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.983718 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "efb50bbbedb9457f88533cc7e32c793a"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.989686 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 02:11:24.993835 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:24.995458 21586 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:11:24.995677 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "efb50bbbedb9457f88533cc7e32c793a"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:11:24.995900 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.027084 21586 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:25.028021 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.028317 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.029543 21586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.031253 21586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.031369 21586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:25.031486 21586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.031569 21586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:25.071050 21586 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:34055
I20240503 02:11:25.071138 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:34055 every 8 connection(s)
I20240503 02:11:25.072502 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:25.080687 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21586
I20240503 02:11:25.080878 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:25.084667 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:25.087035 21713 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:25.087412 21713 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.088649 21713 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:25.090734 21396 ts_manager.cc:194] Registered new tserver with Master: efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:25.091990 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:39107
W20240503 02:11:25.467859 21717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.468183 21717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.468286 21717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.476397 21717 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:25.476544 21717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.476665 21717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:11:25.486070 21717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.487668 21717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.489549 21717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.503150 21723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.502955 21725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.502938 21722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.506945 21717 server_base.cc:1034] running on GCE node
I20240503 02:11:25.507658 21717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.508728 21717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.509955 21717 hybrid_clock.cc:648] HybridClock initialized: now 1714702285509905 us; error 47 us; skew 500 ppm
I20240503 02:11:25.510332 21717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.512890 21717 webserver.cc:460] Webserver started at http://127.20.218.67:39837/ using document root <none> and password file <none>
I20240503 02:11:25.513774 21717 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.513891 21717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.514236 21717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.517722 21717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b98f966b15514275b82deb8c7dadcac9"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-81lq"
I20240503 02:11:25.518600 21717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b98f966b15514275b82deb8c7dadcac9"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-81lq"
I20240503 02:11:25.524590 21717 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.004s
I20240503 02:11:25.528635 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:25.530252 21717 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:11:25.530448 21717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b98f966b15514275b82deb8c7dadcac9"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-81lq"
I20240503 02:11:25.530663 21717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.552893 21717 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:25.553901 21717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.554198 21717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.555271 21717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.556972 21717 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.557066 21717 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:25.557173 21717 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.557240 21717 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:25.562103 21582 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:25.598652 21717 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:45345
I20240503 02:11:25.598711 21843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:45345 every 8 connection(s)
I20240503 02:11:25.600131 21717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:25.607410 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21717
I20240503 02:11:25.607610 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:25.611451 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:25.616393 21844 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:25.616798 21844 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.618001 21844 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:25.619972 21396 ts_manager.cc:194] Registered new tserver with Master: b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:25.621186 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:54343
W20240503 02:11:25.985569 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.985898 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.986001 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.993613 21848 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:25.993755 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.993880 21848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:11:26.003666 21848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:26.004989 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:26.006807 21848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.019711 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.019742 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.019868 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.020612 21848 server_base.cc:1034] running on GCE node
I20240503 02:11:26.021234 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:26.022356 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:26.023605 21848 hybrid_clock.cc:648] HybridClock initialized: now 1714702286023526 us; error 84 us; skew 500 ppm
I20240503 02:11:26.023962 21848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.026438 21848 webserver.cc:460] Webserver started at http://127.20.218.68:33193/ using document root <none> and password file <none>
I20240503 02:11:26.027212 21848 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.027340 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.027659 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.031000 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.031872 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.037914 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.003s
I20240503 02:11:26.042002 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.043666 21848 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:11:26.043867 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.044090 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.065716 21848 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:26.067086 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:26.067426 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.068596 21848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.070291 21848 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.070397 21848 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.070515 21848 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.070588 21848 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.095147 21713 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:26.108331 21848 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:46877
I20240503 02:11:26.108390 21974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:46877 every 8 connection(s)
I20240503 02:11:26.109787 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:26.114934 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21848
I20240503 02:11:26.115190 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:26.118862 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:0
--local_ip_for_outbound_sockets=127.20.218.69
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:26.126196 21975 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:26.126623 21975 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.127759 21975 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:26.130025 21396 ts_manager.cc:194] Registered new tserver with Master: 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:26.131404 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:60159
W20240503 02:11:26.486047 21978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:26.486361 21978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:26.486465 21978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:26.494607 21978 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:26.494761 21978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:26.494885 21978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:11:26.505662 21978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:26.507018 21978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:26.508795 21978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.522384 21984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.522400 21986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.522653 21983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.523768 21978 server_base.cc:1034] running on GCE node
I20240503 02:11:26.524699 21978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:26.525931 21978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:26.527199 21978 hybrid_clock.cc:648] HybridClock initialized: now 1714702286527156 us; error 40 us; skew 500 ppm
I20240503 02:11:26.528021 21978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.530537 21978 webserver.cc:460] Webserver started at http://127.20.218.69:42281/ using document root <none> and password file <none>
I20240503 02:11:26.531309 21978 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.531427 21978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.531740 21978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.534451 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "02fea8896de14021872010185f9408a7"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.535282 21978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "02fea8896de14021872010185f9408a7"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.541272 21978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20240503 02:11:26.546052 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.547891 21978 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:11:26.548081 21978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "02fea8896de14021872010185f9408a7"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:26.548291 21978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.570884 21978 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:26.571736 21978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:26.571990 21978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.572911 21978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.574530 21978 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.574620 21978 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.574704 21978 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.574756 21978 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:26.611833 21978 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:39821
I20240503 02:11:26.611943 22104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:39821 every 8 connection(s)
I20240503 02:11:26.613320 21978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:26.619463 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21978
I20240503 02:11:26.619680 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:26.623819 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:0
--local_ip_for_outbound_sockets=127.20.218.70
--webserver_interface=127.20.218.70
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:26.624953 21844 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:26.629266 22105 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:26.629653 22105 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.630533 22105 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:26.632123 21396 ts_manager.cc:194] Registered new tserver with Master: 02fea8896de14021872010185f9408a7 (127.20.218.69:39821)
I20240503 02:11:26.633304 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:36823
W20240503 02:11:26.998190 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:26.998530 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:26.998677 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:27.006680 22108 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:27.006856 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:27.007025 22108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:11:27.017292 22108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:27.018728 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:27.020560 22108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:27.034572 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.034550 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.034550 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:27.035552 22108 server_base.cc:1034] running on GCE node
I20240503 02:11:27.036262 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:27.037298 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:27.038553 22108 hybrid_clock.cc:648] HybridClock initialized: now 1714702287038494 us; error 51 us; skew 500 ppm
I20240503 02:11:27.038906 22108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:27.041359 22108 webserver.cc:460] Webserver started at http://127.20.218.70:45485/ using document root <none> and password file <none>
I20240503 02:11:27.042147 22108 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:27.042253 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:27.042577 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:27.045367 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "0b5ef04c277e4fb09b5dcb9314371503"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.046275 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "0b5ef04c277e4fb09b5dcb9314371503"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.052153 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:11:27.056727 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.058310 22108 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:11:27.058696 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0b5ef04c277e4fb09b5dcb9314371503"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.059041 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:27.082875 22108 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:27.084373 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:27.084689 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:27.085798 22108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:27.087522 22108 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:27.087618 22108 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.087733 22108 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:27.087800 22108 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.127940 22108 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:36117
I20240503 02:11:27.127990 22234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:36117 every 8 connection(s)
I20240503 02:11:27.129382 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:27.135040 21975 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:27.136421 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 22108
I20240503 02:11:27.136615 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:27.140686 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:0
--local_ip_for_outbound_sockets=127.20.218.71
--webserver_interface=127.20.218.71
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:27.144944 22235 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:27.145453 22235 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:27.146477 22235 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:27.148259 21396 ts_manager.cc:194] Registered new tserver with Master: 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117)
I20240503 02:11:27.149468 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:52039
W20240503 02:11:27.521801 22239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:27.522114 22239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:27.522220 22239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:27.530336 22239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:27.530478 22239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:27.530609 22239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:11:27.542182 22239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:27.543561 22239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:27.545430 22239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:27.558404 22247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.558387 22245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:27.558336 22244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:27.559795 22239 server_base.cc:1034] running on GCE node
I20240503 02:11:27.560484 22239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:27.561575 22239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:27.562892 22239 hybrid_clock.cc:648] HybridClock initialized: now 1714702287562814 us; error 42 us; skew 500 ppm
I20240503 02:11:27.563452 22239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:27.566803 22239 webserver.cc:460] Webserver started at http://127.20.218.71:45945/ using document root <none> and password file <none>
I20240503 02:11:27.567824 22239 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:27.567986 22239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:27.568362 22239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:27.571367 22239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4516a714250344268ce0b17c65356895"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.572321 22239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4516a714250344268ce0b17c65356895"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.578696 22239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 02:11:27.582804 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.584440 22239 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20240503 02:11:27.584642 22239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4516a714250344268ce0b17c65356895"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:27.584867 22239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:27.602574 22239 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:27.603475 22239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:27.603754 22239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:27.604933 22239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:27.606629 22239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:27.606727 22239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.606845 22239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:27.606932 22239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:27.637205 22105 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:27.646023 22239 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:44273
I20240503 02:11:27.646100 22365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:44273 every 8 connection(s)
I20240503 02:11:27.647415 22239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:27.654112 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 22239
I20240503 02:11:27.654343 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:11:27.658211 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.72:0
--local_ip_for_outbound_sockets=127.20.218.72
--webserver_interface=127.20.218.72
--webserver_port=0
--tserver_master_addrs=127.20.218.126:41289
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:27.664528 22366 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:27.664883 22366 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:27.665903 22366 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:27.667723 21396 ts_manager.cc:194] Registered new tserver with Master: 4516a714250344268ce0b17c65356895 (127.20.218.71:44273)
I20240503 02:11:27.668952 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:41041
W20240503 02:11:28.040352 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:28.040719 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:28.040833 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:28.048821 22370 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:28.049001 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:28.049177 22370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.72
I20240503 02:11:28.059629 22370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.218.72
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.72
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:28.061036 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:28.062901 22370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:28.076535 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:28.076660 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:28.076864 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:28.078050 22370 server_base.cc:1034] running on GCE node
I20240503 02:11:28.078869 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:28.079952 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:28.081218 22370 hybrid_clock.cc:648] HybridClock initialized: now 1714702288081165 us; error 57 us; skew 500 ppm
I20240503 02:11:28.081619 22370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:28.084288 22370 webserver.cc:460] Webserver started at http://127.20.218.72:35159/ using document root <none> and password file <none>
I20240503 02:11:28.085129 22370 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:28.085378 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:28.085966 22370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:28.089653 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "5b7a113589d64600a80987859abf90bd"
format_stamp: "Formatted at 2024-05-03 02:11:28 on dist-test-slave-81lq"
I20240503 02:11:28.090724 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "5b7a113589d64600a80987859abf90bd"
format_stamp: "Formatted at 2024-05-03 02:11:28 on dist-test-slave-81lq"
I20240503 02:11:28.096804 22370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240503 02:11:28.101519 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:28.103135 22370 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240503 02:11:28.103314 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5b7a113589d64600a80987859abf90bd"
format_stamp: "Formatted at 2024-05-03 02:11:28 on dist-test-slave-81lq"
I20240503 02:11:28.103543 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:28.128823 22370 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:28.129814 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:28.130108 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:28.131165 22370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:28.132807 22370 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:28.132905 22370 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:28.132992 22370 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:28.133074 22370 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:28.152552 22235 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:28.177112 22370 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.72:39305
I20240503 02:11:28.177255 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.72:39305 every 8 connection(s)
I20240503 02:11:28.178664 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:11:28.181246 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 22370
I20240503 02:11:28.181499 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:11:28.193629 22497 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:28.193966 22497 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:28.194850 22497 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:28.196568 21396 ts_manager.cc:194] Registered new tserver with Master: 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305)
I20240503 02:11:28.197731 21396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.72:48551
I20240503 02:11:28.199309 21353 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:11:28.223410 21353 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:11:28.223714 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21978
I20240503 02:11:28.235985 22100 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:28.321413 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21978
I20240503 02:11:28.338300 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 22108
I20240503 02:11:28.347028 22230 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:28.429211 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 22108
I20240503 02:11:28.452351 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 22239
I20240503 02:11:28.460650 22361 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:28.542327 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 22239
I20240503 02:11:28.558879 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 22370
I20240503 02:11:28.568526 22492 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:28.652853 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 22370
I20240503 02:11:32.423287 21396 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:32.480826 21778 tablet_service.cc:1467] Processing CreateTablet for tablet 5eceeb73c14743d88e84b9a69d3129d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.481966 21778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eceeb73c14743d88e84b9a69d3129d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.485153 21910 tablet_service.cc:1467] Processing CreateTablet for tablet 63989b9f652141379ec1ae9628f73300 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.485142 21515 tablet_service.cc:1467] Processing CreateTablet for tablet 2cae4071aaec4fa4992fb330f51d85fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.486711 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63989b9f652141379ec1ae9628f73300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.486711 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cae4071aaec4fa4992fb330f51d85fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.489763 21516 tablet_service.cc:1467] Processing CreateTablet for tablet 5eceeb73c14743d88e84b9a69d3129d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.490312 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eceeb73c14743d88e84b9a69d3129d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.489867 21517 tablet_service.cc:1467] Processing CreateTablet for tablet 63989b9f652141379ec1ae9628f73300 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.491264 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63989b9f652141379ec1ae9628f73300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.480195 21779 tablet_service.cc:1467] Processing CreateTablet for tablet 63989b9f652141379ec1ae9628f73300 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.492660 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63989b9f652141379ec1ae9628f73300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.494962 21648 tablet_service.cc:1467] Processing CreateTablet for tablet 5eceeb73c14743d88e84b9a69d3129d7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.495975 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 2cae4071aaec4fa4992fb330f51d85fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.497006 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cae4071aaec4fa4992fb330f51d85fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.497879 21909 tablet_service.cc:1467] Processing CreateTablet for tablet 2cae4071aaec4fa4992fb330f51d85fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=d645570f595c4475953474a1b8b0242f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:32.498538 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cae4071aaec4fa4992fb330f51d85fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.500612 21648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eceeb73c14743d88e84b9a69d3129d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:32.508430 22528 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:32.514297 22528 tablet_bootstrap.cc:654] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.519336 22528 log.cc:824] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:32.520601 22527 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:32.524787 22527 tablet_bootstrap.cc:654] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.525137 22529 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:32.526907 22527 log.cc:824] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:32.533027 22531 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:32.533695 22527 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:32.534119 22527 ts_tablet_manager.cc:1367] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.000s
I20240503 02:11:32.534262 22529 tablet_bootstrap.cc:654] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.535099 22528 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:32.535588 22529 log.cc:824] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:32.537488 22528 ts_tablet_manager.cc:1367] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.002s
I20240503 02:11:32.540509 22529 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:32.540952 22529 ts_tablet_manager.cc:1367] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.004s
I20240503 02:11:32.545449 22527 raft_consensus.cc:348] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.546044 22527 raft_consensus.cc:374] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.546190 22527 raft_consensus.cc:729] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.546976 22527 consensus_queue.cc:260] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.553375 22529 raft_consensus.cc:348] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.554081 22529 raft_consensus.cc:374] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.554260 22529 raft_consensus.cc:729] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.555262 22529 consensus_queue.cc:260] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.557045 22531 tablet_bootstrap.cc:654] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.557598 22527 ts_tablet_manager.cc:1398] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.023s	user 0.022s	sys 0.000s
I20240503 02:11:32.558112 22527 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:32.560359 22527 tablet_bootstrap.cc:654] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.563310 22531 log.cc:824] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:32.570621 22529 ts_tablet_manager.cc:1398] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.008s
I20240503 02:11:32.571187 22529 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:32.573441 22529 tablet_bootstrap.cc:654] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.576134 22527 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:32.576464 22527 ts_tablet_manager.cc:1367] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.012s
I20240503 02:11:32.577863 22527 raft_consensus.cc:348] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.578199 22527 raft_consensus.cc:374] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.578353 22527 raft_consensus.cc:729] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.578676 22527 consensus_queue.cc:260] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.580340 22527 ts_tablet_manager.cc:1398] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:11:32.580372 22531 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:32.580680 22531 ts_tablet_manager.cc:1367] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.048s	user 0.007s	sys 0.004s
I20240503 02:11:32.580539 22528 raft_consensus.cc:348] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.581087 22528 raft_consensus.cc:374] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.581265 22528 raft_consensus.cc:729] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
W20240503 02:11:32.581456 21714 tablet.cc:2369] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:32.582149 22528 consensus_queue.cc:260] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.587738 22529 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:32.587967 22529 ts_tablet_manager.cc:1367] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.017s	user 0.001s	sys 0.003s
I20240503 02:11:32.589442 22529 raft_consensus.cc:348] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.589697 22529 raft_consensus.cc:374] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.589785 22529 raft_consensus.cc:729] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.590086 22529 consensus_queue.cc:260] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.591328 22528 ts_tablet_manager.cc:1398] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.053s	user 0.011s	sys 0.010s
I20240503 02:11:32.592000 22528 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:32.592555 22535 raft_consensus.cc:482] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:32.592795 22535 raft_consensus.cc:504] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.594130 22528 tablet_bootstrap.cc:654] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.594967 22535 leader_election.cc:288] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:32.602154 22529 ts_tablet_manager.cc:1398] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20240503 02:11:32.602185 22531 raft_consensus.cc:348] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.602908 22531 raft_consensus.cc:374] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.606315 22531 raft_consensus.cc:729] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.607393 22531 consensus_queue.cc:260] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
W20240503 02:11:32.608886 21845 tablet.cc:2369] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:32.611917 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cae4071aaec4fa4992fb330f51d85fc" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:32.612641 21930 raft_consensus.cc:2454] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 0.
I20240503 02:11:32.613755 21603 leader_election.cc:302] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:32.614351 22535 raft_consensus.cc:2781] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:32.614516 22535 raft_consensus.cc:482] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:32.614624 22535 raft_consensus.cc:3037] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.616230 22528 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:32.616564 22528 ts_tablet_manager.cc:1367] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.025s	user 0.003s	sys 0.006s
W20240503 02:11:32.618140 21976 tablet.cc:2369] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:32.618714 22528 raft_consensus.cc:348] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.618950 22528 raft_consensus.cc:374] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.618820 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cae4071aaec4fa4992fb330f51d85fc" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" is_pre_election: true
I20240503 02:11:32.619041 22528 raft_consensus.cc:729] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.619416 22528 consensus_queue.cc:260] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
W20240503 02:11:32.620177 21603 leader_election.cc:341] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:32.620514 22535 raft_consensus.cc:504] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.621095 22531 ts_tablet_manager.cc:1398] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.040s	user 0.012s	sys 0.006s
I20240503 02:11:32.621666 22531 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:32.622056 22528 ts_tablet_manager.cc:1398] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:11:32.623548 22531 tablet_bootstrap.cc:654] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.625890 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cae4071aaec4fa4992fb330f51d85fc" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
W20240503 02:11:32.626752 21603 leader_election.cc:341] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:32.627041 22535 leader_election.cc:288] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:32.627573 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cae4071aaec4fa4992fb330f51d85fc" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:32.627815 21930 raft_consensus.cc:3037] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.629129 22531 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:32.629492 22531 ts_tablet_manager.cc:1367] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.006s
I20240503 02:11:32.631191 22531 raft_consensus.cc:348] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.631495 22531 raft_consensus.cc:374] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.631702 22531 raft_consensus.cc:729] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.631975 21930 raft_consensus.cc:2454] T 2cae4071aaec4fa4992fb330f51d85fc P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 1.
I20240503 02:11:32.632205 22531 consensus_queue.cc:260] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.632746 21603 leader_election.cc:302] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:32.633009 22531 ts_tablet_manager.cc:1398] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:32.633205 22535 raft_consensus.cc:2781] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:32.633394 22531 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:32.635181 22535 raft_consensus.cc:686] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Becoming Leader. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Running, Role: LEADER
I20240503 02:11:32.635701 22531 tablet_bootstrap.cc:654] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:32.635838 22535 consensus_queue.cc:237] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.638360 22531 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:32.638592 22531 ts_tablet_manager.cc:1367] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:11:32.639472 22531 raft_consensus.cc:348] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.639827 22531 raft_consensus.cc:374] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:32.639938 22531 raft_consensus.cc:729] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:32.640213 22531 consensus_queue.cc:260] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:32.641131 22531 ts_tablet_manager.cc:1398] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:11:32.642311 21395 catalog_manager.cc:5484] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a reported cstate change: term changed from 0 to 1, leader changed from <none> to efb50bbbedb9457f88533cc7e32c793a (127.20.218.66). New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:32.651726 22543 raft_consensus.cc:482] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:32.652009 22543 raft_consensus.cc:504] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.653705 22543 leader_election.cc:288] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:32.664956 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:32.665593 21799 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:32.666257 21472 leader_election.cc:302] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:32.666664 22543 raft_consensus.cc:2781] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:32.666795 22543 raft_consensus.cc:482] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:32.666900 22543 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.668787 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:32.669147 21930 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:32.670174 22543 raft_consensus.cc:504] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.670889 22543 leader_election.cc:288] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:32.671598 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:32.671841 21930 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.671778 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:32.672106 21799 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.673949 21930 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:32.674373 21472 leader_election.cc:302] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939; no voters: 
I20240503 02:11:32.674656 22543 raft_consensus.cc:2781] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:32.675220 22543 raft_consensus.cc:686] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:32.675843 22543 consensus_queue.cc:237] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:32.676968 21799 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:32.681671 21396 catalog_manager.cc:5484] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:32.798269 21583 tablet.cc:2369] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:32.823367 22543 raft_consensus.cc:482] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:32.823643 22543 raft_consensus.cc:504] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.824458 22543 leader_election.cc:288] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:32.825148 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:32.825503 21799 raft_consensus.cc:2454] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:32.825964 21472 leader_election.cc:302] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:32.826354 22543 raft_consensus.cc:2781] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:32.826473 22543 raft_consensus.cc:482] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:32.827275 22543 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.829751 22543 raft_consensus.cc:504] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.830518 22543 leader_election.cc:288] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:32.831142 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:32.831409 21799 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.831849 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:32.831880 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:32.832458 21667 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:32.833789 21799 raft_consensus.cc:2454] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:32.834214 21472 leader_election.cc:302] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:32.834522 22543 raft_consensus.cc:2781] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:32.834710 22543 raft_consensus.cc:686] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:32.835004 22543 consensus_queue.cc:237] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:32.835512 21667 raft_consensus.cc:2454] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:32.838549 21396 catalog_manager.cc:5484] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:33.028191 21537 raft_consensus.cc:3037] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:33.032732 22535 consensus_queue.cc:1035] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.042578 22557 consensus_queue.cc:1035] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.061910 22543 consensus_queue.cc:1035] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.068109 22561 consensus_queue.cc:1035] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:11:33.255970 22561 consensus_queue.cc:1035] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:33.261140 22543 consensus_queue.cc:1035] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:33.287053 22553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:33.287411 22553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:33.295156 22553 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 275.49 ms
  time per row: 0.137745 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:34.680049 21390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:34.690654 21910 tablet_service.cc:1467] Processing CreateTablet for tablet 1c11c73db59748b59b8fba963f543fbb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.690855 21779 tablet_service.cc:1467] Processing CreateTablet for tablet 1c11c73db59748b59b8fba963f543fbb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.691192 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c11c73db59748b59b8fba963f543fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.691304 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c11c73db59748b59b8fba963f543fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.691776 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 1c11c73db59748b59b8fba963f543fbb (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.692214 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c11c73db59748b59b8fba963f543fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.692284 21648 tablet_service.cc:1467] Processing CreateTablet for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.692659 21648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99da2b3ee7cc414e8642cf26d46a3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.696033 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 7e357b085a8e48d9901ac648dd89e31f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.696597 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e357b085a8e48d9901ac648dd89e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.701052 21909 tablet_service.cc:1467] Processing CreateTablet for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.701408 22621 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:34.701808 21516 tablet_service.cc:1467] Processing CreateTablet for tablet 7e357b085a8e48d9901ac648dd89e31f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.702394 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e357b085a8e48d9901ac648dd89e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.702682 21778 tablet_service.cc:1467] Processing CreateTablet for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.703256 21778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99da2b3ee7cc414e8642cf26d46a3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.704118 22621 tablet_bootstrap.cc:654] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.705964 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 7e357b085a8e48d9901ac648dd89e31f (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:34.706633 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e357b085a8e48d9901ac648dd89e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.710347 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99da2b3ee7cc414e8642cf26d46a3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.711807 22620 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:34.713795 22620 tablet_bootstrap.cc:654] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.726835 22620 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:34.727892 22620 ts_tablet_manager.cc:1367] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.006s
I20240503 02:11:34.729487 22619 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:34.729391 22620 raft_consensus.cc:348] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.729637 22620 raft_consensus.cc:374] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.729753 22620 raft_consensus.cc:729] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.730067 22620 consensus_queue.cc:260] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.733309 22624 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:34.734539 22619 tablet_bootstrap.cc:654] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.735124 22624 tablet_bootstrap.cc:654] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.737138 22621 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:34.737514 22621 ts_tablet_manager.cc:1367] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.036s	user 0.005s	sys 0.011s
I20240503 02:11:34.738710 22621 raft_consensus.cc:348] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.739125 22621 raft_consensus.cc:374] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.739289 22621 raft_consensus.cc:729] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.739683 22621 consensus_queue.cc:260] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.740518 22621 ts_tablet_manager.cc:1398] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:11:34.740881 22621 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:34.742743 22621 tablet_bootstrap.cc:654] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.744171 22620 ts_tablet_manager.cc:1398] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.004s
I20240503 02:11:34.744568 22620 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:34.746305 22619 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:34.746555 22619 ts_tablet_manager.cc:1367] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.005s
I20240503 02:11:34.746723 22620 tablet_bootstrap.cc:654] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.747494 22619 raft_consensus.cc:348] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.747884 22619 raft_consensus.cc:374] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.748056 22619 raft_consensus.cc:729] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.748549 22619 consensus_queue.cc:260] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.750401 22624 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:34.750536 22621 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:34.750612 22624 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.008s
I20240503 02:11:34.750768 22621 ts_tablet_manager.cc:1367] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.003s
I20240503 02:11:34.750980 22620 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:34.751127 22620 ts_tablet_manager.cc:1367] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.003s
I20240503 02:11:34.751395 22624 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.751622 22624 raft_consensus.cc:374] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.751529 22621 raft_consensus.cc:348] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.751721 22624 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.751741 22621 raft_consensus.cc:374] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.751812 22621 raft_consensus.cc:729] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.751854 22620 raft_consensus.cc:348] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.752102 22620 raft_consensus.cc:374] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.752226 22620 raft_consensus.cc:729] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.752108 22621 consensus_queue.cc:260] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.752233 22624 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.752705 22620 consensus_queue.cc:260] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.753463 22621 ts_tablet_manager.cc:1398] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:34.753829 22621 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:34.754241 22624 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:11:34.754947 22619 ts_tablet_manager.cc:1398] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.001s
I20240503 02:11:34.755056 22620 ts_tablet_manager.cc:1398] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:11:34.755335 22619 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:34.755702 22621 tablet_bootstrap.cc:654] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.757197 22619 tablet_bootstrap.cc:654] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.762251 22619 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:34.762529 22621 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:34.762553 22619 ts_tablet_manager.cc:1367] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.007s	user 0.007s	sys 0.000s
I20240503 02:11:34.762784 22621 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.003s
I20240503 02:11:34.763384 22619 raft_consensus.cc:348] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.763583 22619 raft_consensus.cc:374] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.763659 22619 raft_consensus.cc:729] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.763552 22621 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.763756 22621 raft_consensus.cc:374] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.763880 22621 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.763913 22619 consensus_queue.cc:260] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.764106 22621 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.764472 22619 ts_tablet_manager.cc:1398] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:34.764681 22621 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:34.764818 22619 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:34.766633 22619 tablet_bootstrap.cc:654] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.768797 22619 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:34.768972 22619 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:11:34.769883 22619 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.770099 22619 raft_consensus.cc:374] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.770196 22619 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.770443 22619 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.770928 22619 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:34.815532 22625 raft_consensus.cc:482] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.815827 22625 raft_consensus.cc:504] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.817636 22625 leader_election.cc:288] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.822063 22630 raft_consensus.cc:482] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.822305 22630 raft_consensus.cc:504] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.824362 22630 leader_election.cc:288] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.832185 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c11c73db59748b59b8fba963f543fbb" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:34.832600 21667 raft_consensus.cc:2454] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:34.833225 21864 leader_election.cc:302] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:34.833719 22625 raft_consensus.cc:2781] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.833848 22625 raft_consensus.cc:482] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.833948 22625 raft_consensus.cc:3037] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.837148 22625 raft_consensus.cc:504] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.839589 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c11c73db59748b59b8fba963f543fbb" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:34.839826 21667 raft_consensus.cc:3037] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.841910 22625 leader_election.cc:288] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.842813 21667 raft_consensus.cc:2454] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:34.843494 21864 leader_election.cc:302] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:34.844110 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99da2b3ee7cc414e8642cf26d46a3cc3" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:34.844365 21668 raft_consensus.cc:2454] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 0.
I20240503 02:11:34.844903 21733 leader_election.cc:302] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:34.845520 22630 raft_consensus.cc:2781] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.845670 22630 raft_consensus.cc:482] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.845777 22630 raft_consensus.cc:3037] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.846439 22625 raft_consensus.cc:2781] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.848862 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c11c73db59748b59b8fba963f543fbb" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:34.849131 21799 raft_consensus.cc:2454] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:34.849905 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c11c73db59748b59b8fba963f543fbb" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:34.850131 21799 raft_consensus.cc:3037] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.850456 22630 raft_consensus.cc:504] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.852355 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99da2b3ee7cc414e8642cf26d46a3cc3" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:34.852636 21667 raft_consensus.cc:3037] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.853240 21799 raft_consensus.cc:2454] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:34.853605 22630 leader_election.cc:288] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.854048 22625 raft_consensus.cc:686] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Becoming Leader. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:11:34.854561 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99da2b3ee7cc414e8642cf26d46a3cc3" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:34.854636 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99da2b3ee7cc414e8642cf26d46a3cc3" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:34.854807 21930 raft_consensus.cc:2454] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 0.
I20240503 02:11:34.854709 22625 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:34.856457 21667 raft_consensus.cc:2454] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:34.856902 21733 leader_election.cc:302] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 4a634cdcef0a4c3ba054106d3c65f9ae
I20240503 02:11:34.857278 22630 raft_consensus.cc:2781] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.858063 22630 raft_consensus.cc:686] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:34.858533 22630 consensus_queue.cc:237] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:34.859237 21390 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68). New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:34.862625 21395 catalog_manager.cc:5484] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:34.912835 22597 raft_consensus.cc:482] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:34.913043 22597 raft_consensus.cc:504] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.913864 22597 leader_election.cc:288] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.914680 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:34.914706 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:34.914925 21799 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:34.914944 21667 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:34.915477 21471 leader_election.cc:302] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:34.915771 22597 raft_consensus.cc:2781] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:34.915886 22597 raft_consensus.cc:482] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:34.915961 22597 raft_consensus.cc:3037] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.917958 22597 raft_consensus.cc:504] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.918536 22597 leader_election.cc:288] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:34.919307 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:34.919275 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:34.919512 21667 raft_consensus.cc:3037] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.919577 21799 raft_consensus.cc:3037] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.921532 21667 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:34.921872 21799 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:34.922052 21471 leader_election.cc:302] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:34.922451 22597 raft_consensus.cc:2781] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.922883 22597 raft_consensus.cc:686] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:34.923542 22597 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:34.926505 21395 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:35.239912 22625 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.249471 22642 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.298357 21929 raft_consensus.cc:3037] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.301508 22630 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.307940 22644 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:35.331439 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.331812 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:35.339733 22648 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:11:35.483875 22597 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:35.507061 22691 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 256.548 ms
  time per row: 0.128274 ms
I20240503 02:11:35.760808 21390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:35.771739 21909 tablet_service.cc:1467] Processing CreateTablet for tablet aeb0c27e2f5f4649bce1259462637cab (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.771739 21516 tablet_service.cc:1467] Processing CreateTablet for tablet aeb0c27e2f5f4649bce1259462637cab (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.771777 21647 tablet_service.cc:1467] Processing CreateTablet for tablet aeb0c27e2f5f4649bce1259462637cab (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.772473 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb0c27e2f5f4649bce1259462637cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.772559 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb0c27e2f5f4649bce1259462637cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.772557 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb0c27e2f5f4649bce1259462637cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.778040 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2bb56fb72428cbbebfa69ba154306 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.778596 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2bb56fb72428cbbebfa69ba154306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.778847 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 2a236b927e104bf48a35c975a5507c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.779014 21910 tablet_service.cc:1467] Processing CreateTablet for tablet 2a236b927e104bf48a35c975a5507c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.779273 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a236b927e104bf48a35c975a5507c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.779469 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a236b927e104bf48a35c975a5507c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.781949 21908 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2bb56fb72428cbbebfa69ba154306 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.782424 21908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2bb56fb72428cbbebfa69ba154306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.786185 22715 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:35.786404 21515 tablet_service.cc:1467] Processing CreateTablet for tablet 2a236b927e104bf48a35c975a5507c5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.786809 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a236b927e104bf48a35c975a5507c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.788121 22715 tablet_bootstrap.cc:654] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.789894 21517 tablet_service.cc:1467] Processing CreateTablet for tablet 33e2bb56fb72428cbbebfa69ba154306 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:35.790431 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e2bb56fb72428cbbebfa69ba154306. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:35.801069 22718 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:35.801472 22717 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:35.801596 22715 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:35.801924 22715 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.000s
I20240503 02:11:35.803314 22718 tablet_bootstrap.cc:654] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.803180 22715 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.803701 22715 raft_consensus.cc:374] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.803882 22715 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.803985 22717 tablet_bootstrap.cc:654] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.804733 22715 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.805871 22719 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:35.807973 22715 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240503 02:11:35.808357 22715 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:35.808637 22719 tablet_bootstrap.cc:654] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.811220 22715 tablet_bootstrap.cc:654] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.811605 22719 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:35.811841 22719 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:11:35.812189 22717 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:35.812258 22718 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:35.812361 22717 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240503 02:11:35.812458 22718 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240503 02:11:35.812851 22719 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.813158 22719 raft_consensus.cc:374] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.813305 22719 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.813393 22718 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.813441 22717 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.813673 22718 raft_consensus.cc:374] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.813751 22717 raft_consensus.cc:374] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.813871 22717 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.813751 22719 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.813900 22718 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.814293 22718 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.814620 22719 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:35.815461 22717 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.815930 22718 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:11:35.816320 22718 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:35.816944 22715 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:35.817245 22715 ts_tablet_manager.cc:1367] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240503 02:11:35.818207 22718 tablet_bootstrap.cc:654] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.818325 22715 raft_consensus.cc:348] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.818668 22715 raft_consensus.cc:374] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.818826 22715 raft_consensus.cc:729] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.819164 22715 consensus_queue.cc:260] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.819941 22715 ts_tablet_manager.cc:1398] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:35.821779 22718 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:35.822041 22718 ts_tablet_manager.cc:1367] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 02:11:35.822391 22717 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240503 02:11:35.822760 22717 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:35.823158 22718 raft_consensus.cc:348] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.823501 22718 raft_consensus.cc:374] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.823683 22718 raft_consensus.cc:729] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.824039 22718 consensus_queue.cc:260] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.824754 22717 tablet_bootstrap.cc:654] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.824790 22718 ts_tablet_manager.cc:1398] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:35.825186 22718 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:35.826819 22718 tablet_bootstrap.cc:654] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.832214 22717 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:35.832242 22718 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:35.832399 22718 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.003s
I20240503 02:11:35.832422 22717 ts_tablet_manager.cc:1367] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240503 02:11:35.833230 22717 raft_consensus.cc:348] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.833252 22718 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.833523 22718 raft_consensus.cc:374] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.833519 22717 raft_consensus.cc:374] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.833614 22717 raft_consensus.cc:729] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.833623 22718 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.833874 22718 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.833894 22717 consensus_queue.cc:260] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.834465 22717 ts_tablet_manager.cc:1398] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:35.834457 22718 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:35.835353 22717 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:35.837576 22717 tablet_bootstrap.cc:654] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:35.840440 22717 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:35.840626 22717 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:11:35.841279 22717 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.841483 22717 raft_consensus.cc:374] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:35.841579 22717 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:35.841791 22717 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:35.842291 22717 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:11:35.960283 22597 raft_consensus.cc:482] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:35.960489 22597 raft_consensus.cc:504] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.961108 22597 leader_election.cc:288] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:35.961797 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:35.961844 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:35.962029 21929 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:35.962062 21798 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:35.962538 21472 leader_election.cc:302] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939; no voters: 
I20240503 02:11:35.962823 22597 raft_consensus.cc:2781] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:35.962968 22597 raft_consensus.cc:482] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:35.963063 22597 raft_consensus.cc:3037] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.964975 22597 raft_consensus.cc:504] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.965651 22597 leader_election.cc:288] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:35.966349 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:35.966554 21798 raft_consensus.cc:3037] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.966485 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:35.966812 21929 raft_consensus.cc:3037] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:35.968793 21798 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:35.969205 21929 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:35.969357 21472 leader_election.cc:302] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:35.969727 22597 raft_consensus.cc:2781] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:35.969909 22597 raft_consensus.cc:686] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:35.970239 22597 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:35.973093 21390 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:36.013439 22557 raft_consensus.cc:482] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:36.013659 22557 raft_consensus.cc:504] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.014286 22557 leader_election.cc:288] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:36.015018 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb0c27e2f5f4649bce1259462637cab" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:36.015106 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb0c27e2f5f4649bce1259462637cab" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" is_pre_election: true
I20240503 02:11:36.015295 21929 raft_consensus.cc:2454] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 0.
I20240503 02:11:36.015431 21537 raft_consensus.cc:2454] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 0.
I20240503 02:11:36.015838 21603 leader_election.cc:302] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:36.016165 22557 raft_consensus.cc:2781] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:36.016278 22557 raft_consensus.cc:482] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:36.016386 22557 raft_consensus.cc:3037] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.018383 22557 raft_consensus.cc:504] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.019032 22557 leader_election.cc:288] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:36.019665 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb0c27e2f5f4649bce1259462637cab" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:36.019722 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb0c27e2f5f4649bce1259462637cab" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
I20240503 02:11:36.019876 21929 raft_consensus.cc:3037] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.020051 21537 raft_consensus.cc:3037] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.022428 21929 raft_consensus.cc:2454] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 1.
I20240503 02:11:36.022555 21537 raft_consensus.cc:2454] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 1.
I20240503 02:11:36.022895 21603 leader_election.cc:302] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:36.023283 22557 raft_consensus.cc:2781] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:36.023308 22597 raft_consensus.cc:482] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:36.023491 22597 raft_consensus.cc:504] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.023941 22557 raft_consensus.cc:686] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Becoming Leader. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Running, Role: LEADER
I20240503 02:11:36.024363 22597 leader_election.cc:288] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:36.024307 22557 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.024976 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:36.025184 21667 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:36.025439 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:36.025666 21929 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:36.025763 21471 leader_election.cc:302] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:36.026173 22597 raft_consensus.cc:2781] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:36.026536 22597 raft_consensus.cc:482] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:36.026734 22597 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.028445 21395 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a reported cstate change: term changed from 0 to 1, leader changed from <none> to efb50bbbedb9457f88533cc7e32c793a (127.20.218.66). New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:36.028975 22597 raft_consensus.cc:504] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.029857 22597 leader_election.cc:288] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:36.030077 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:36.030247 21929 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.030208 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:36.030406 21667 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:36.032466 21929 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:36.032949 21472 leader_election.cc:302] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939; no voters: 
I20240503 02:11:36.032949 21667 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:36.033322 22597 raft_consensus.cc:2781] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:36.033564 22597 raft_consensus.cc:686] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:36.033845 22597 consensus_queue.cc:237] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:36.036653 21395 catalog_manager.cc:5484] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:36.448431 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:36.448801 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:36.457159 22732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:36.480737 22557 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:36.487526 22729 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:36.489938 22597 consensus_queue.cc:1035] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 02:11:36.497009 22691 consensus_queue.cc:1035] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:36.509495 22691 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:36.517444 22597 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 230.897 ms
  time per row: 0.115448 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:11:37.811762 21389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:37.823129 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 323dc3d2761e4114a2f87832d2510ef4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.823235 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 323dc3d2761e4114a2f87832d2510ef4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.823640 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.823662 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.824086 21515 tablet_service.cc:1467] Processing CreateTablet for tablet 323dc3d2761e4114a2f87832d2510ef4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.824072 21909 tablet_service.cc:1467] Processing CreateTablet for tablet 9e03e6f7855b48b5a35fcd84a9279714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.824489 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e03e6f7855b48b5a35fcd84a9279714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.824494 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.830983 21646 tablet_service.cc:1467] Processing CreateTablet for tablet 143591b9348249c7836e2f121eb1b073 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.831437 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143591b9348249c7836e2f121eb1b073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.834417 21516 tablet_service.cc:1467] Processing CreateTablet for tablet 9e03e6f7855b48b5a35fcd84a9279714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.834779 21778 tablet_service.cc:1467] Processing CreateTablet for tablet 9e03e6f7855b48b5a35fcd84a9279714 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.835145 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e03e6f7855b48b5a35fcd84a9279714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.835448 21778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e03e6f7855b48b5a35fcd84a9279714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.842322 22801 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:37.844604 22801 tablet_bootstrap.cc:654] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.846424 21910 tablet_service.cc:1467] Processing CreateTablet for tablet 143591b9348249c7836e2f121eb1b073 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.847095 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143591b9348249c7836e2f121eb1b073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.844596 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 143591b9348249c7836e2f121eb1b073 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.849931 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143591b9348249c7836e2f121eb1b073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.862807 22799 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:37.865135 22799 tablet_bootstrap.cc:654] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.865613 22803 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:37.867591 22803 tablet_bootstrap.cc:654] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.872733 22800 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:37.875614 22800 tablet_bootstrap.cc:654] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.877234 22799 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:37.877534 22799 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.007s
I20240503 02:11:37.878526 22799 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.878757 22799 raft_consensus.cc:374] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.878870 22799 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.879256 22803 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:37.879169 22799 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.879420 22803 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.010s
I20240503 02:11:37.879724 22799 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:11:37.879510 22800 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:37.880017 22800 ts_tablet_manager.cc:1367] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:11:37.880195 22803 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.880365 22803 raft_consensus.cc:374] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.880483 22803 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.880506 22801 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:37.880653 22801 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.019s
I20240503 02:11:37.880748 22803 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.880860 22800 raft_consensus.cc:348] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.881135 22800 raft_consensus.cc:374] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.881278 22800 raft_consensus.cc:729] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.881367 22803 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240503 02:11:37.881618 22801 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.881628 22800 consensus_queue.cc:260] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.881798 22803 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:37.881929 22801 raft_consensus.cc:374] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.882050 22801 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.882297 22801 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.883499 22803 tablet_bootstrap.cc:654] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.885375 22801 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:11:37.885793 22801 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:37.887117 22800 ts_tablet_manager.cc:1398] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:37.886329 22803 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:37.887562 22803 ts_tablet_manager.cc:1367] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20240503 02:11:37.888605 22803 raft_consensus.cc:348] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.888871 22803 raft_consensus.cc:374] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.888988 22803 raft_consensus.cc:729] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.889154 22800 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:37.889289 22803 consensus_queue.cc:260] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.890121 22803 ts_tablet_manager.cc:1398] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:37.890478 22799 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:37.891491 22801 tablet_bootstrap.cc:654] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.892261 22799 tablet_bootstrap.cc:654] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.894464 22799 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:37.894640 22799 ts_tablet_manager.cc:1367] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20240503 02:11:37.896428 22799 raft_consensus.cc:348] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.896735 22799 raft_consensus.cc:374] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.896864 22799 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.897183 22799 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.897852 22799 ts_tablet_manager.cc:1398] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:37.899976 22800 tablet_bootstrap.cc:654] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.900888 22801 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:37.901109 22801 ts_tablet_manager.cc:1367] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.009s
I20240503 02:11:37.902698 22801 raft_consensus.cc:348] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.902935 22801 raft_consensus.cc:374] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.903009 22801 raft_consensus.cc:729] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.903245 22801 consensus_queue.cc:260] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.903779 22801 ts_tablet_manager.cc:1398] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240503 02:11:37.907639 22801 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:37.908046 22800 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:37.908308 22800 ts_tablet_manager.cc:1367] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.000s
I20240503 02:11:37.909205 22800 raft_consensus.cc:348] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.909499 22800 raft_consensus.cc:374] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.909600 22800 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.909647 22801 tablet_bootstrap.cc:654] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.909933 22800 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.913946 22800 ts_tablet_manager.cc:1398] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:37.915563 22801 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:37.915803 22801 ts_tablet_manager.cc:1367] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240503 02:11:37.916740 22801 raft_consensus.cc:348] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.916966 22801 raft_consensus.cc:374] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.917114 22801 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.917408 22801 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.917963 22801 ts_tablet_manager.cc:1398] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:37.918711 22642 raft_consensus.cc:482] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.918911 22642 raft_consensus.cc:504] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.919814 22642 leader_election.cc:288] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:37.921026 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e03e6f7855b48b5a35fcd84a9279714" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:37.921283 21798 raft_consensus.cc:2454] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:37.925983 21865 leader_election.cc:302] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:37.926421 22642 raft_consensus.cc:2781] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.926530 22642 raft_consensus.cc:482] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.926651 22642 raft_consensus.cc:3037] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.929738 22642 raft_consensus.cc:504] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.930471 22642 leader_election.cc:288] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:37.931051 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e03e6f7855b48b5a35fcd84a9279714" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:37.931277 21798 raft_consensus.cc:3037] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.933215 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e03e6f7855b48b5a35fcd84a9279714" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" is_pre_election: true
I20240503 02:11:37.933296 21536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e03e6f7855b48b5a35fcd84a9279714" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
I20240503 02:11:37.933583 21537 raft_consensus.cc:2454] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:37.933688 21798 raft_consensus.cc:2454] T 9e03e6f7855b48b5a35fcd84a9279714 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:37.934262 21865 leader_election.cc:302] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:37.934558 22642 raft_consensus.cc:2781] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.934963 22642 raft_consensus.cc:686] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Becoming Leader. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:11:37.935278 22642 consensus_queue.cc:237] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:37.938198 21390 catalog_manager.cc:5484] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68). New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:37.977768 22642 raft_consensus.cc:482] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.978052 22642 raft_consensus.cc:504] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.978842 22642 leader_election.cc:288] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:37.979769 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:37.979847 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:37.980093 21667 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:37.980191 21798 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 0.
I20240503 02:11:37.980670 21864 leader_election.cc:302] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:37.980971 22642 raft_consensus.cc:2781] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.981113 22642 raft_consensus.cc:482] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.981192 22642 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.983201 22642 raft_consensus.cc:504] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.983858 22642 leader_election.cc:288] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:37.984458 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:37.984649 21798 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.984640 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:37.984918 21667 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.986785 21798 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:37.987250 21667 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:37.987336 21865 leader_election.cc:302] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:37.987742 22642 raft_consensus.cc:2781] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.988013 22642 raft_consensus.cc:686] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Becoming Leader. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:11:37.988327 22642 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:37.991242 21390 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68). New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:38.082495 22777 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:38.082731 22777 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:38.083472 22777 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:38.084060 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:38.084080 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:38.084321 21667 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:38.084334 21798 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:38.084789 21471 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:38.085126 22777 raft_consensus.cc:2781] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:38.085275 22777 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:38.085398 22777 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.087738 22777 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:38.088402 22777 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:38.088981 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:38.088959 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:38.089195 21667 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.089226 21798 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.092144 21798 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:38.092159 21667 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:38.092835 21471 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:38.093202 22777 raft_consensus.cc:2781] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:38.093621 22777 raft_consensus.cc:686] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:38.093936 22777 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:38.097385 21390 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:38.361668 22642 consensus_queue.cc:1035] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:38.368484 21536 raft_consensus.cc:3037] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:38.373028 22814 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:38.374285 22814 consensus_queue.cc:1035] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:38.380074 22823 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:38.534034 22819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:38.534730 22819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:38.545621 22819 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:38.551224 22817 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:38.559536 22817 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
W20240503 02:11:38.917109 21865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:39543, user_credentials={real_user=slave}} blocked reactor thread for 48660us
W20240503 02:11:38.928808 21864 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34055, user_credentials={real_user=slave}} blocked reactor thread for 63173.6us

INSERT report
    rows total: 2000
    time total: 345.722 ms
  time per row: 0.172861 ms
I20240503 02:11:39.082443 21394 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52782:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:11:39.099845 21515 tablet_service.cc:1467] Processing CreateTablet for tablet c047b5ecea1a4e2da5c2322d5eed0373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.100503 21515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047b5ecea1a4e2da5c2322d5eed0373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.101702 21646 tablet_service.cc:1467] Processing CreateTablet for tablet c047b5ecea1a4e2da5c2322d5eed0373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.101761 21777 tablet_service.cc:1467] Processing CreateTablet for tablet 35b131174c34438d8f72fb2473f47dfa (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.102165 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047b5ecea1a4e2da5c2322d5eed0373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.102365 21777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.106007 21778 tablet_service.cc:1467] Processing CreateTablet for tablet daa192225bab430484e3e822a63d6316 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.106573 21778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa192225bab430484e3e822a63d6316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.106984 21647 tablet_service.cc:1467] Processing CreateTablet for tablet 35b131174c34438d8f72fb2473f47dfa (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.107347 21516 tablet_service.cc:1467] Processing CreateTablet for tablet 35b131174c34438d8f72fb2473f47dfa (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.107503 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.107368 21517 tablet_service.cc:1467] Processing CreateTablet for tablet daa192225bab430484e3e822a63d6316 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.107893 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa192225bab430484e3e822a63d6316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.109629 21516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.114897 21909 tablet_service.cc:1467] Processing CreateTablet for tablet c047b5ecea1a4e2da5c2322d5eed0373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.115571 21909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c047b5ecea1a4e2da5c2322d5eed0373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.117296 21910 tablet_service.cc:1467] Processing CreateTablet for tablet daa192225bab430484e3e822a63d6316 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:39.118103 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa192225bab430484e3e822a63d6316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:39.119597 22883 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:39.123059 22883 tablet_bootstrap.cc:654] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.127703 22884 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:39.131386 22884 tablet_bootstrap.cc:654] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.133248 22882 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:39.135182 22883 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:39.135478 22882 tablet_bootstrap.cc:654] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.135540 22883 ts_tablet_manager.cc:1367] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.006s
I20240503 02:11:39.136832 22883 raft_consensus.cc:348] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.137862 22883 raft_consensus.cc:374] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.138051 22883 raft_consensus.cc:729] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.139087 22883 consensus_queue.cc:260] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.139994 22883 ts_tablet_manager.cc:1398] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:11:39.141672 22884 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:39.141980 22884 ts_tablet_manager.cc:1367] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.000s
I20240503 02:11:39.143306 22884 raft_consensus.cc:348] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.143713 22884 raft_consensus.cc:374] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.143878 22884 raft_consensus.cc:729] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.144357 22884 consensus_queue.cc:260] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.151831 22882 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:39.152606 22884 ts_tablet_manager.cc:1398] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240503 02:11:39.152972 22882 ts_tablet_manager.cc:1367] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.003s
I20240503 02:11:39.154449 22882 raft_consensus.cc:348] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.157982 22883 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Bootstrap starting.
I20240503 02:11:39.159292 22884 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:39.160413 22883 tablet_bootstrap.cc:654] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.162261 22882 raft_consensus.cc:374] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.162653 22882 raft_consensus.cc:729] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.163301 22882 consensus_queue.cc:260] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.163817 22630 raft_consensus.cc:482] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.164059 22630 raft_consensus.cc:504] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.165176 22630 leader_election.cc:288] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:39.167749 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa192225bab430484e3e822a63d6316" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:39.174769 22883 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: No bootstrap required, opened a new log
I20240503 02:11:39.174993 22883 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.002s
I20240503 02:11:39.175745 22883 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.175953 22883 raft_consensus.cc:374] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.176031 22883 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.176296 22883 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.176915 22883 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:39.177474 22882 ts_tablet_manager.cc:1398] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.024s	user 0.003s	sys 0.000s
I20240503 02:11:39.178159 22885 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:39.178695 22882 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Bootstrap starting.
I20240503 02:11:39.180536 22882 tablet_bootstrap.cc:654] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.180574 22885 tablet_bootstrap.cc:654] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.178447 22884 tablet_bootstrap.cc:654] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
W20240503 02:11:39.182186 21734 leader_election.cc:341] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:11:39.188426 22884 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:39.189538 22884 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.004s
I20240503 02:11:39.190459 22884 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.190675 22884 raft_consensus.cc:374] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.190608 21536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa192225bab430484e3e822a63d6316" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" is_pre_election: true
I20240503 02:11:39.190943 21536 raft_consensus.cc:2454] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 0.
I20240503 02:11:39.191720 21734 leader_election.cc:302] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 4a634cdcef0a4c3ba054106d3c65f9ae
I20240503 02:11:39.192102 22630 raft_consensus.cc:2781] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.192253 22630 raft_consensus.cc:482] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.190963 22884 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.192369 22630 raft_consensus.cc:3037] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.192606 22884 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.193240 22884 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240503 02:11:39.193971 22884 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:11:39.195960 22884 tablet_bootstrap.cc:654] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.197189 22882 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: No bootstrap required, opened a new log
I20240503 02:11:39.197418 22882 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.000s
I20240503 02:11:39.198318 22882 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.198609 22882 raft_consensus.cc:374] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.198705 22882 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.199002 22882 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.199638 22882 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:11:39.200857 22885 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:39.201108 22885 ts_tablet_manager.cc:1367] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.000s
I20240503 02:11:39.202069 22885 raft_consensus.cc:348] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.202385 22885 raft_consensus.cc:374] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.202517 22885 raft_consensus.cc:729] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.202860 22885 consensus_queue.cc:260] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.203766 22885 ts_tablet_manager.cc:1398] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:39.204198 22885 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae: Bootstrap starting.
I20240503 02:11:39.205543 22884 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: No bootstrap required, opened a new log
I20240503 02:11:39.205786 22884 ts_tablet_manager.cc:1367] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.000s
I20240503 02:11:39.206246 22885 tablet_bootstrap.cc:654] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:39.206771 22884 raft_consensus.cc:348] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.207049 22884 raft_consensus.cc:374] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.207186 22884 raft_consensus.cc:729] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.207494 22884 consensus_queue.cc:260] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.208353 22884 ts_tablet_manager.cc:1398] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:11:39.212873 22885 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae: No bootstrap required, opened a new log
I20240503 02:11:39.213084 22885 ts_tablet_manager.cc:1367] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240503 02:11:39.213850 22885 raft_consensus.cc:348] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.214047 22885 raft_consensus.cc:374] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:39.214134 22885 raft_consensus.cc:729] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Initialized, Role: FOLLOWER
I20240503 02:11:39.214443 22885 consensus_queue.cc:260] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.215355 22885 ts_tablet_manager.cc:1398] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:11:39.249621 22630 raft_consensus.cc:504] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.250666 22630 leader_election.cc:288] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:39.251303 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa192225bab430484e3e822a63d6316" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:39.251413 21536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa192225bab430484e3e822a63d6316" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
I20240503 02:11:39.251570 21930 raft_consensus.cc:3037] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.251643 21536 raft_consensus.cc:3037] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.349939 21930 raft_consensus.cc:2454] T daa192225bab430484e3e822a63d6316 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:39.349900 21536 raft_consensus.cc:2454] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:39.350608 21734 leader_election.cc:302] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:39.351083 22630 raft_consensus.cc:2781] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.351521 22630 raft_consensus.cc:686] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [term 1 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:39.351850 22630 consensus_queue.cc:237] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:39.356133 21388 catalog_manager.cc:5484] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:39.390565 22817 raft_consensus.cc:482] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.390801 22817 raft_consensus.cc:504] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.391501 22817 leader_election.cc:288] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:39.392168 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:39.392199 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:39.392422 21666 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:39.392422 21799 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:39.392928 21472 leader_election.cc:302] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:39.393236 22817 raft_consensus.cc:2781] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.393692 22817 raft_consensus.cc:482] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.393833 22817 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.413087 22898 raft_consensus.cc:482] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:39.413496 22898 raft_consensus.cc:504] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.414528 22898 leader_election.cc:288] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:39.415287 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:39.415400 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:39.415558 21930 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:39.415678 21666 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 0.
I20240503 02:11:39.416111 21472 leader_election.cc:302] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939; no voters: 
I20240503 02:11:39.416507 22898 raft_consensus.cc:2781] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:39.416637 22898 raft_consensus.cc:482] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:39.416715 22898 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.450522 22817 raft_consensus.cc:504] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.451599 22817 leader_election.cc:288] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:39.452507 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:39.452782 21799 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.452678 21666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:39.452986 21666 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.453231 22898 raft_consensus.cc:504] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.454084 22898 leader_election.cc:288] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:39.455792 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:39.456198 21930 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.456312 21666 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:39.456924 21471 leader_election.cc:302] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:39.457409 22898 raft_consensus.cc:2781] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.457569 22898 raft_consensus.cc:686] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:39.457777 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:39.457895 22898 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:39.458036 21667 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:39.461622 21388 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:39.551928 21930 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:39.551928 21799 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:39.552598 21667 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a1eb67d644b6cac7d2c95ab4ac939 in term 1.
I20240503 02:11:39.553030 21472 leader_election.cc:302] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939; no voters: 
I20240503 02:11:39.553496 22898 raft_consensus.cc:2781] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:39.553741 22898 raft_consensus.cc:686] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Leader. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:39.554088 22898 consensus_queue.cc:237] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:39.557763 21388 catalog_manager.cc:5484] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: term changed from 0 to 1, leader changed from <none> to 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:39.837728 22630 consensus_queue.cc:1035] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.856700 22907 consensus_queue.cc:1035] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.909826 22898 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:39.915071 22899 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:40.017586 22902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.017966 22902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:40.020056 22899 consensus_queue.cc:1035] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:40.026134 22817 consensus_queue.cc:1035] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:40.029247 22902 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 256.698 ms
  time per row: 0.128349 ms
I20240503 02:11:40.458523 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:39821
--local_ip_for_outbound_sockets=127.20.218.69
--tserver_master_addrs=127.20.218.126:41289
--webserver_port=42281
--webserver_interface=127.20.218.69
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:11:40.849370 22965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:40.849766 22965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:40.849980 22965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:40.858615 22965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:40.858803 22965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:40.858984 22965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:11:40.869644 22965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:39821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=42281
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:40.871147 22965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:40.872961 22965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:40.887413 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.887384 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:40.887477 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:40.888106 22965 server_base.cc:1034] running on GCE node
I20240503 02:11:40.888808 22965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:40.889988 22965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:40.891252 22965 hybrid_clock.cc:648] HybridClock initialized: now 1714702300891207 us; error 39 us; skew 500 ppm
I20240503 02:11:40.891619 22965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:40.895181 22965 webserver.cc:460] Webserver started at http://127.20.218.69:42281/ using document root <none> and password file <none>
I20240503 02:11:40.896080 22965 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:40.896214 22965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:40.903069 22965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:11:40.909715 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:40.911862 22965 fs_manager.cc:725] Time spent opening block manager: real 0.007s	user 0.007s	sys 0.000s
I20240503 02:11:40.912122 22965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "02fea8896de14021872010185f9408a7"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-81lq"
I20240503 02:11:40.913400 22965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:40.948799 22965 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:40.949760 22965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:40.950076 22965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:40.951135 22965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:40.952817 22965 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:40.952901 22965 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:40.952996 22965 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:40.953032 22965 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:40.991619 22965 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:39821
I20240503 02:11:40.991686 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:39821 every 8 connection(s)
I20240503 02:11:40.993227 22965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:41.003621 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 22965
I20240503 02:11:41.004321 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:36117
--local_ip_for_outbound_sockets=127.20.218.70
--tserver_master_addrs=127.20.218.126:41289
--webserver_port=45485
--webserver_interface=127.20.218.70
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:41.012387 23092 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:41.012725 23092 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:41.013886 23092 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:41.016371 21390 ts_manager.cc:194] Re-registered known tserver with Master: 02fea8896de14021872010185f9408a7 (127.20.218.69:39821)
I20240503 02:11:41.017844 21390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:39789
W20240503 02:11:41.413369 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.413697 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.413825 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.421905 23096 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.422050 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.422209 23096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:11:41.433099 23096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:36117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=45485
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.434649 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.436772 23096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.452939 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.452939 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.453621 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.453822 23096 server_base.cc:1034] running on GCE node
I20240503 02:11:41.455070 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.457057 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.458336 23096 hybrid_clock.cc:648] HybridClock initialized: now 1714702301458298 us; error 81 us; skew 500 ppm
I20240503 02:11:41.458762 23096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.462172 23096 webserver.cc:460] Webserver started at http://127.20.218.70:45485/ using document root <none> and password file <none>
I20240503 02:11:41.463081 23096 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.463270 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.469828 23096 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.007s	sys 0.000s
I20240503 02:11:41.473115 23111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:41.474829 23096 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:11:41.475044 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "0b5ef04c277e4fb09b5dcb9314371503"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:41.476668 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.492591 23096 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:41.493608 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.493928 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.495047 23096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:41.496728 23096 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:41.496812 23096 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:41.497468 23096 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:41.497534 23096 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s	user 0.000s	sys 0.001s
I20240503 02:11:41.538261 23096 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:36117
I20240503 02:11:41.538275 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:36117 every 8 connection(s)
I20240503 02:11:41.539980 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:41.540840 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23096
I20240503 02:11:41.541857 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:44273
--local_ip_for_outbound_sockets=127.20.218.71
--tserver_master_addrs=127.20.218.126:41289
--webserver_port=45945
--webserver_interface=127.20.218.71
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:41.580884 23224 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:41.581291 23224 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:41.582334 23224 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:41.584012 21390 ts_manager.cc:194] Re-registered known tserver with Master: 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117)
I20240503 02:11:41.584851 21390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:46631
W20240503 02:11:41.979449 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.979846 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.980006 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.989615 23228 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.989836 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.990036 23228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:11:42.004141 23228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:44273
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=45945
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.005826 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.007880 23228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:11:42.022487 23092 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
W20240503 02:11:42.024910 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.024963 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.025130 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.027148 23228 server_base.cc:1034] running on GCE node
I20240503 02:11:42.027926 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.029214 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.030578 23228 hybrid_clock.cc:648] HybridClock initialized: now 1714702302030485 us; error 102 us; skew 500 ppm
I20240503 02:11:42.031008 23228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.033826 23228 webserver.cc:460] Webserver started at http://127.20.218.71:45945/ using document root <none> and password file <none>
I20240503 02:11:42.034703 23228 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.034857 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.043010 23228 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240503 02:11:42.047971 23242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:42.050469 23228 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:11:42.050776 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4516a714250344268ce0b17c65356895"
format_stamp: "Formatted at 2024-05-03 02:11:27 on dist-test-slave-81lq"
I20240503 02:11:42.051791 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.075316 23228 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:42.076661 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.077092 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.078713 23228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.081291 23228 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.081486 23228 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20240503 02:11:42.081641 23228 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.081722 23228 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:42.129376 23228 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:44273
I20240503 02:11:42.129487 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:44273 every 8 connection(s)
I20240503 02:11:42.131139 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:42.139789 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23228
I20240503 02:11:42.140648 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.72:39305
--local_ip_for_outbound_sockets=127.20.218.72
--tserver_master_addrs=127.20.218.126:41289
--webserver_port=35159
--webserver_interface=127.20.218.72
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240503 02:11:42.150652 23355 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:42.151024 23355 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.151997 23355 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:42.154417 21393 ts_manager.cc:194] Re-registered known tserver with Master: 4516a714250344268ce0b17c65356895 (127.20.218.71:44273)
I20240503 02:11:42.155875 21393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:39157
W20240503 02:11:42.579341 23359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.579689 23359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.579797 23359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:11:42.587826 23224 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
W20240503 02:11:42.588016 23359 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:42.588192 23359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.588408 23359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.72
I20240503 02:11:42.599594 23359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.72:39305
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.218.72
--webserver_port=35159
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.72
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.601166 23359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.603197 23359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.618417 23364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.619071 23367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.619151 23365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.619511 23359 server_base.cc:1034] running on GCE node
I20240503 02:11:42.620436 23359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.621800 23359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.623092 23359 hybrid_clock.cc:648] HybridClock initialized: now 1714702302623020 us; error 60 us; skew 500 ppm
I20240503 02:11:42.623534 23359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.626581 23359 webserver.cc:460] Webserver started at http://127.20.218.72:35159/ using document root <none> and password file <none>
I20240503 02:11:42.627439 23359 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.627599 23359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.635562 23359 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:11:42.640452 23373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:42.642302 23359 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:11:42.642527 23359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "5b7a113589d64600a80987859abf90bd"
format_stamp: "Formatted at 2024-05-03 02:11:28 on dist-test-slave-81lq"
I20240503 02:11:42.643488 23359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.683660 23359 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:11:42.684604 23359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.684921 23359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.686638 23359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.688351 23359 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.688450 23359 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:42.688591 23359 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.688647 23359 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:11:42.729737 23359 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.72:39305
I20240503 02:11:42.729768 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.72:39305 every 8 connection(s)
I20240503 02:11:42.731974 23359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:11:42.736838 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23359
I20240503 02:11:42.752198 23486 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:11:42.752593 23486 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.753659 23486 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:11:42.755625 21393 ts_manager.cc:194] Re-registered known tserver with Master: 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305)
I20240503 02:11:42.756971 21393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.72:55239
I20240503 02:11:43.159894 23355 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:43.368901 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:43.369575 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:43.392287 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:43.408998 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:43.434666 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:43.447885 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:43.459836 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:43.479565 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:43.760586 23486 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
I20240503 02:11:44.097311 21393 ts_manager.cc:292] Set tserver state for 652a1eb67d644b6cac7d2c95ab4ac939 to MAINTENANCE_MODE
W20240503 02:11:45.570511 23560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:45.570889 23560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:45.579048 23560 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:45.777671 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.868484 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:45.869081 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.870437 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.877400 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.909785 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:45.912285 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:45.914222 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:45.916241 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.964319 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:45.992770 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:46.017578 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.041518 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:46.050936 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.078251 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.078922 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.383358 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.389885 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.395061 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.409796 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:46.414789 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.448310 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.448810 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:46.475353 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.721158 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.747489 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.752804 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.779944 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:46.803777 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:11:46.818317 21471 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34055, user_credentials={real_user=slave}} blocked reactor thread for 46769.6us
I20240503 02:11:46.839951 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:46.853009 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:46.887775 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:11:46.918743 23672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:46.919288 23672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:46.933368 23672 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:47.002480 21537 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:47.004984 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 652a1eb67d644b6cac7d2c95ab4ac939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:11:47.005023 21667 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 652a1eb67d644b6cac7d2c95ab4ac939: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:11:47.006199 23556 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:47.006596 23724 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:47.010501 23724 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:47.011348 21667 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:47.015130 21799 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:47.019762 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:47.035864 21537 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:47.044082 21667 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 652a1eb67d644b6cac7d2c95ab4ac939: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:47.045812 23724 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:47.051354 23724 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:11:47.057057 21667 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:11:47.059403 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 652a1eb67d644b6cac7d2c95ab4ac939: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:47.060424 23556 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:47.062250 21799 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:11:47.066345 21382 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 35b131174c34438d8f72fb2473f47dfa with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:11:47.068612 21391 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 reported cstate change: config changed from index 4 to 5, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:47.080756 21472 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 -> Peer 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117): Couldn't send request to peer 0b5ef04c277e4fb09b5dcb9314371503. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b131174c34438d8f72fb2473f47dfa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:47.090044 21537 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:32868
I20240503 02:11:47.090359 21537 raft_consensus.cc:595] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:47.167172 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:47.179800 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:47.180457 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:47.205294 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:47.219040 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:47.232393 21517 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:47.265195 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:47.302939 21646 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:47.647958 23798 ts_tablet_manager.cc:902] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Initiating tablet copy from peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:47.650774 23798 tablet_copy_client.cc:308] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:39543
I20240503 02:11:47.660351 21557 tablet_copy_service.cc:140] P 652a1eb67d644b6cac7d2c95ab4ac939: Received BeginTabletCopySession request for tablet 35b131174c34438d8f72fb2473f47dfa from peer 0b5ef04c277e4fb09b5dcb9314371503 ({username='slave'} at 127.20.218.70:34243)
I20240503 02:11:47.660637 21557 tablet_copy_service.cc:161] P 652a1eb67d644b6cac7d2c95ab4ac939: Beginning new tablet copy session on tablet 35b131174c34438d8f72fb2473f47dfa from peer 0b5ef04c277e4fb09b5dcb9314371503 at {username='slave'} at 127.20.218.70:34243: session id = 0b5ef04c277e4fb09b5dcb9314371503-35b131174c34438d8f72fb2473f47dfa
I20240503 02:11:47.663695 21557 tablet_copy_source_session.cc:205] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:47.666886 23724 raft_consensus.cc:979] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: : Instructing follower efb50bbbedb9457f88533cc7e32c793a to start an election
I20240503 02:11:47.667124 23556 raft_consensus.cc:1067] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Signalling peer efb50bbbedb9457f88533cc7e32c793a to start an election
I20240503 02:11:47.668265 21667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
 from {username='slave'} at 127.20.218.65:43033
I20240503 02:11:47.668572 21667 raft_consensus.cc:482] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:47.668716 21667 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:47.669165 23798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:47.672086 21667 raft_consensus.cc:504] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:47.675660 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
I20240503 02:11:47.675966 21537 raft_consensus.cc:3032] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:47.676115 21537 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Running, Role: LEADER
I20240503 02:11:47.676559 21537 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:47.677111 21537 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:47.680393 21537 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 2.
I20240503 02:11:47.681169 21603 leader_election.cc:302] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:47.681897 22774 raft_consensus.cc:2781] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:47.685356 21667 leader_election.cc:288] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:47.685462 23798 tablet_copy_client.cc:783] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:47.685894 23798 tablet_copy_client.cc:647] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:47.686869 22774 raft_consensus.cc:686] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Becoming Leader. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Running, Role: LEADER
I20240503 02:11:47.687534 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:47.687418 22774 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:47.687790 21799 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:47.690099 23802 tablet_copy_client.cc:1002] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Injecting latency into file download: 1500
I20240503 02:11:47.690765 21799 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 2.
I20240503 02:11:47.692339 21390 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to efb50bbbedb9457f88533cc7e32c793a (127.20.218.66). New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:47.861160 23792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:47.861567 23792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:47.869637 23792 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:11:48.049255 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:48.049511 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:48.065429 21537 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:11:48.066221 22774 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:11:48.082422 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:48.083118 21909 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:48.112846 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:48.123514 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:11:48.123713 21777 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:11:48.124286 22774 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:11:48.131685 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:11:48.153298 21517 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:11:48.157807 21865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45345, user_credentials={real_user=slave}} blocked reactor thread for 103694us
W20240503 02:11:48.170856 21603 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a -> Peer 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117): Couldn't send request to peer 0b5ef04c277e4fb09b5dcb9314371503. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
Runtime error: ksck discovered errors
I20240503 02:11:48.376260 23558 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21455
I20240503 02:11:48.383920 21577 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:48.526031 23558 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21455
W20240503 02:11:48.555519 21734 connection.cc:535] server connection from 127.20.218.65:34317 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:48.555599 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.218.65:39543 (error 108)
W20240503 02:11:48.557793 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.557984 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.721772 21603 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.797297 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:48.799145 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:48.936705 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:48.938222 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:49.190723 23798 tablet_copy_client.cc:523] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:49.194301 23798 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap starting.
I20240503 02:11:49.204459 23798 log.cc:824] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:49.238942 23798 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:49.239583 23798 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap complete.
I20240503 02:11:49.240049 23798 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Time spent bootstrapping tablet: real 0.046s	user 0.043s	sys 0.000s
I20240503 02:11:49.245625 23798 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:49.246105 23798 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Initialized, Role: LEARNER
I20240503 02:11:49.246635 23798 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:11:49.248265 23798 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.004s
W20240503 02:11:49.250335 23798 tablet_copy_client.cc:1108] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.474480 23900 raft_consensus.cc:482] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.474722 23900 raft_consensus.cc:504] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:49.475533 23900 leader_election.cc:288] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:49.476578 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
W20240503 02:11:49.477134 21603 leader_election.cc:334] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.477447 21603 leader_election.cc:302] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efb50bbbedb9457f88533cc7e32c793a; no voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.477711 23900 raft_consensus.cc:2726] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.546521 23916 raft_consensus.cc:482] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.546751 23916 raft_consensus.cc:504] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.547439 23916 leader_election.cc:288] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.548069 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
W20240503 02:11:49.549079 21865 leader_election.cc:334] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.549240 21865 leader_election.cc:302] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae; no voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9
I20240503 02:11:49.549593 23916 raft_consensus.cc:2726] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.550658 23900 raft_consensus.cc:482] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.550810 23900 raft_consensus.cc:504] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.551445 23900 leader_election.cc:288] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:49.552335 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
W20240503 02:11:49.552752 21603 leader_election.cc:334] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.553004 21603 leader_election.cc:302] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efb50bbbedb9457f88533cc7e32c793a; no voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.553287 23900 raft_consensus.cc:2726] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.578395 23179 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Advancing to term 2
I20240503 02:11:49.588694 23916 raft_consensus.cc:482] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.588874 23916 raft_consensus.cc:504] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.589562 23916 leader_election.cc:288] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.590018 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
W20240503 02:11:49.590879 21865 leader_election.cc:334] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.591017 21865 leader_election.cc:302] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae; no voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9
I20240503 02:11:49.591264 23916 raft_consensus.cc:2726] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.604993 23901 raft_consensus.cc:482] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.605197 23901 raft_consensus.cc:504] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:49.606060 23901 leader_election.cc:288] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:49.606990 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
W20240503 02:11:49.607786 21603 leader_election.cc:334] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.607901 21603 leader_election.cc:302] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9
I20240503 02:11:49.608175 23901 raft_consensus.cc:2726] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.658986 23920 raft_consensus.cc:482] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.659180 23920 raft_consensus.cc:504] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.659866 23920 leader_election.cc:288] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.660626 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:49.660943 21929 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:49.661309 21734 leader_election.cc:302] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:11:49.661621 23920 raft_consensus.cc:2781] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.661748 23920 raft_consensus.cc:482] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
W20240503 02:11:49.661754 21734 leader_election.cc:334] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.661892 23920 raft_consensus.cc:3037] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.663971 23920 raft_consensus.cc:504] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.664532 23920 leader_election.cc:288] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Requested vote from peers 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.665053 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a236b927e104bf48a35c975a5507c5e" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:49.665284 21929 raft_consensus.cc:3037] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.665971 21734 leader_election.cc:334] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.667357 21929 raft_consensus.cc:2454] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 2.
I20240503 02:11:49.667846 21734 leader_election.cc:302] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.668228 23920 raft_consensus.cc:2781] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.668394 23920 raft_consensus.cc:686] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:49.668704 23920 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.671669 21396 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:49.689134 23916 raft_consensus.cc:482] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.689383 23916 raft_consensus.cc:504] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:49.690227 23916 leader_election.cc:288] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:49.690889 23901 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.690809 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:49.691119 21667 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:49.691112 23901 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.691759 23901 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.691759 21864 leader_election.cc:302] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
W20240503 02:11:49.691942 21865 leader_election.cc:334] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.692184 23916 raft_consensus.cc:2781] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.692315 23916 raft_consensus.cc:482] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.692413 23916 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.692445 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
W20240503 02:11:49.693313 21603 leader_election.cc:334] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.693470 21603 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939, b98f966b15514275b82deb8c7dadcac9
I20240503 02:11:49.694031 23901 raft_consensus.cc:2726] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.694823 23916 raft_consensus.cc:504] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:49.695408 23916 leader_election.cc:288] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: Requested vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:49.696043 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e2bb56fb72428cbbebfa69ba154306" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:49.696307 21667 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.696743 21865 leader_election.cc:334] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.698319 21667 raft_consensus.cc:2454] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 2.
I20240503 02:11:49.698770 21864 leader_election.cc:302] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.699142 23916 raft_consensus.cc:2781] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.699302 23916 raft_consensus.cc:686] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 LEADER]: Becoming Leader. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:11:49.699595 23916 consensus_queue.cc:237] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:49.703191 21396 catalog_manager.cc:5484] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68). New cstate: current_term: 2 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:49.743534 23920 raft_consensus.cc:482] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.743758 23920 raft_consensus.cc:504] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.744419 23920 leader_election.cc:288] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:49.745445 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" is_pre_election: true
I20240503 02:11:49.745692 21929 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
W20240503 02:11:49.745870 21734 leader_election.cc:334] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.746102 21734 leader_election.cc:302] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.746376 23920 raft_consensus.cc:2781] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.746455 23920 raft_consensus.cc:482] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.746557 23920 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.748747 23920 raft_consensus.cc:504] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.749487 23920 leader_election.cc:288] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Requested vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:11:49.750442 21929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63989b9f652141379ec1ae9628f73300" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:11:49.750681 21929 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.751034 21734 leader_election.cc:334] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.753082 21929 raft_consensus.cc:2454] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 2.
I20240503 02:11:49.753638 21734 leader_election.cc:302] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, b98f966b15514275b82deb8c7dadcac9; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.754019 23920 raft_consensus.cc:2781] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.754217 23920 raft_consensus.cc:686] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:49.754498 23920 consensus_queue.cc:237] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.757284 21396 catalog_manager.cc:5484] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:49.783355 23920 raft_consensus.cc:482] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.783561 23920 raft_consensus.cc:504] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:49.784229 23920 leader_election.cc:288] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.784893 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:49.785112 21667 raft_consensus.cc:2454] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:49.785604 21733 leader_election.cc:302] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:11:49.785887 23920 raft_consensus.cc:2781] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.786126 23920 raft_consensus.cc:482] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.786243 23920 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.786347 21734 leader_election.cc:334] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.788287 23920 raft_consensus.cc:504] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:49.788992 23920 leader_election.cc:288] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.789631 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eceeb73c14743d88e84b9a69d3129d7" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:49.789883 21667 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.790585 21734 leader_election.cc:334] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.792271 21667 raft_consensus.cc:2454] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 2.
I20240503 02:11:49.792688 21733 leader_election.cc:302] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.793004 23920 raft_consensus.cc:2781] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.793150 23920 raft_consensus.cc:686] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:49.793562 23920 consensus_queue.cc:237] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:11:49.796663 21396 catalog_manager.cc:5484] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:49.820466 23920 raft_consensus.cc:482] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.820782 23920 raft_consensus.cc:504] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.821686 23920 leader_election.cc:288] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.822211 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
W20240503 02:11:49.823275 21734 leader_election.cc:334] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.823431 21734 leader_election.cc:302] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9; no voters: 652a1eb67d644b6cac7d2c95ab4ac939, efb50bbbedb9457f88533cc7e32c793a
I20240503 02:11:49.823808 23920 raft_consensus.cc:2726] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:11:49.840970 23916 raft_consensus.cc:482] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.841204 23916 raft_consensus.cc:504] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.842048 23916 leader_election.cc:288] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:49.842720 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:49.842998 21667 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 1.
I20240503 02:11:49.843533 21864 leader_election.cc:302] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 
W20240503 02:11:49.843765 21865 leader_election.cc:334] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.843899 23916 raft_consensus.cc:2781] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.844002 23916 raft_consensus.cc:482] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.844111 23916 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.846285 23916 raft_consensus.cc:504] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.846985 23916 leader_election.cc:288] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: Requested vote from peers 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543), efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:49.847548 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c047b5ecea1a4e2da5c2322d5eed0373" candidate_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:49.847779 21667 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:49.848351 21865 leader_election.cc:334] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.850209 21667 raft_consensus.cc:2454] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a634cdcef0a4c3ba054106d3c65f9ae in term 2.
I20240503 02:11:49.850885 21864 leader_election.cc:302] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.851292 23916 raft_consensus.cc:2781] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.851487 23916 raft_consensus.cc:686] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 LEADER]: Becoming Leader. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:11:49.851783 23916 consensus_queue.cc:237] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:11:49.854734 21396 catalog_manager.cc:5484] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68). New cstate: current_term: 2 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:49.876482 23901 raft_consensus.cc:482] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.876698 23901 raft_consensus.cc:504] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.877422 23901 leader_election.cc:288] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:49.878273 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9" is_pre_election: true
I20240503 02:11:49.878590 21799 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 1.
I20240503 02:11:49.879397 21603 leader_election.cc:302] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
W20240503 02:11:49.879662 21603 leader_election.cc:334] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.879700 23901 raft_consensus.cc:2781] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:49.879835 23901 raft_consensus.cc:482] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:49.879933 23901 raft_consensus.cc:3037] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.882120 23901 raft_consensus.cc:504] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.882822 23901 leader_election.cc:288] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
W20240503 02:11:49.884404 21603 leader_election.cc:334] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:49.884307 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e357b085a8e48d9901ac648dd89e31f" candidate_uuid: "efb50bbbedb9457f88533cc7e32c793a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:11:49.884753 21799 raft_consensus.cc:3037] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:49.887355 21799 raft_consensus.cc:2454] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb50bbbedb9457f88533cc7e32c793a in term 2.
I20240503 02:11:49.887908 21603 leader_election.cc:302] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:49.888250 23901 raft_consensus.cc:2781] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:49.888411 23901 raft_consensus.cc:686] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Becoming Leader. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Running, Role: LEADER
I20240503 02:11:49.888674 23901 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:49.891779 21396 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to efb50bbbedb9457f88533cc7e32c793a (127.20.218.66). New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:50.004957 23900 raft_consensus.cc:1050] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 to VOTER
I20240503 02:11:50.006480 23900 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:11:50.009836 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:11:50.010010 23179 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 02:11:50.010285 21603 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.010643 23900 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:50.010944 23950 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:50.013655 23950 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:11:50.014307 21799 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:11:50.017645 23179 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:11:50.018343 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 5 to 7, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:50.023620 21667 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:11:50.026242 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:50.026881 23948 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:50.027565 23178 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:50.028432 23920 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:50.028595 23920 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:50.029367 23920 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:50.030339 23950 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:50.030182 23901 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:11:50.030270 21799 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
W20240503 02:11:50.031981 21734 leader_election.cc:334] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:50.033303 21381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35b131174c34438d8f72fb2473f47dfa with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:11:50.034785 23178 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:11:50.035028 21390 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:50.037613 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a" is_pre_election: true
I20240503 02:11:50.037892 21667 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 1.
I20240503 02:11:50.038385 21733 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:50.038700 23920 raft_consensus.cc:2781] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:11:50.038815 23920 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 652a1eb67d644b6cac7d2c95ab4ac939)
I20240503 02:11:50.038936 23920 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:50.040776 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:50.041710 23920 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:50.042472 23920 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:11:50.043131 21667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:11:50.043380 21667 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:11:50.043576 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:50.044189 21734 leader_election.cc:334] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:50.046550 21667 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 2.
I20240503 02:11:50.047093 21733 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 652a1eb67d644b6cac7d2c95ab4ac939
I20240503 02:11:50.047427 23920 raft_consensus.cc:2781] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:50.047601 23920 raft_consensus.cc:686] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:11:50.047890 23920 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:11:50.051232 21390 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 1 to 2, leader changed from 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:50.105815 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.126644 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.127611 23920 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:50.157064 21667 raft_consensus.cc:1261] T 33e2bb56fb72428cbbebfa69ba154306 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.157898 23916 consensus_queue.cc:1035] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:11:50.164980 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.188670 21929 raft_consensus.cc:1261] T 63989b9f652141379ec1ae9628f73300 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.189425 23958 consensus_queue.cc:1035] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:11:50.194957 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:50.202476 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.265491 21667 raft_consensus.cc:1261] T 5eceeb73c14743d88e84b9a69d3129d7 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.266208 23919 consensus_queue.cc:1035] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:50.348063 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.348971 23900 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:11:50.354158 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.379890 21667 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:11:50.382103 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:11:50.383057 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.383167 23901 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:50.386797 23900 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:50.387790 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } }
I20240503 02:11:50.392760 21390 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:50.397826 21667 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:11:50.400424 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:50.402580 21667 raft_consensus.cc:1261] T c047b5ecea1a4e2da5c2322d5eed0373 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.402755 23900 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:50.403453 23965 consensus_queue.cc:1035] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:11:50.404026 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.406507 23901 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:11:50.407626 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:11:50.411684 21381 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aeb0c27e2f5f4649bce1259462637cab with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:11:50.412343 21602 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 4516a714250344268ce0b17c65356895 (127.20.218.71:44273): Couldn't send request to peer 4516a714250344268ce0b17c65356895. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeb0c27e2f5f4649bce1259462637cab. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.414474 21392 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 4 to 5, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
W20240503 02:11:50.426616 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:50.438310 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:50.508263 21667 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:11:50.508945 23958 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:11:50.805960 23991 ts_tablet_manager.cc:902] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Initiating tablet copy from peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:50.808079 23991 tablet_copy_client.cc:308] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34055
I20240503 02:11:50.815616 21688 tablet_copy_service.cc:140] P efb50bbbedb9457f88533cc7e32c793a: Received BeginTabletCopySession request for tablet aeb0c27e2f5f4649bce1259462637cab from peer 4516a714250344268ce0b17c65356895 ({username='slave'} at 127.20.218.71:43003)
I20240503 02:11:50.815920 21688 tablet_copy_service.cc:161] P efb50bbbedb9457f88533cc7e32c793a: Beginning new tablet copy session on tablet aeb0c27e2f5f4649bce1259462637cab from peer 4516a714250344268ce0b17c65356895 at {username='slave'} at 127.20.218.71:43003: session id = 4516a714250344268ce0b17c65356895-aeb0c27e2f5f4649bce1259462637cab
I20240503 02:11:50.819110 21688 tablet_copy_source_session.cc:205] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:50.824385 23991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb0c27e2f5f4649bce1259462637cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.842355 23991 tablet_copy_client.cc:783] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:50.842952 23991 tablet_copy_client.cc:647] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:50.866895 23995 tablet_copy_client.cc:1002] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: tablet copy: Injecting latency into file download: 1500
W20240503 02:11:50.927966 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:51.258147 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:51.304216 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:51.438704 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:52.328567 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:52.367822 23991 tablet_copy_client.cc:523] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:52.372995 23991 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Bootstrap starting.
I20240503 02:11:52.384369 23991 log.cc:824] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:52.421286 23991 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:52.421860 23991 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Bootstrap complete.
I20240503 02:11:52.422312 23991 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Time spent bootstrapping tablet: real 0.050s	user 0.042s	sys 0.008s
I20240503 02:11:52.428108 23991 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:11:52.428650 23991 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4516a714250344268ce0b17c65356895, State: Initialized, Role: LEARNER
I20240503 02:11:52.429188 23991 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:11:52.430925 23991 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240503 02:11:52.431968 21688 tablet_copy_service.cc:342] P efb50bbbedb9457f88533cc7e32c793a: Request end of tablet copy session 4516a714250344268ce0b17c65356895-aeb0c27e2f5f4649bce1259462637cab received from {username='slave'} at 127.20.218.71:43003
I20240503 02:11:52.432134 21688 tablet_copy_service.cc:433] P efb50bbbedb9457f88533cc7e32c793a: ending tablet copy session 4516a714250344268ce0b17c65356895-aeb0c27e2f5f4649bce1259462637cab on tablet aeb0c27e2f5f4649bce1259462637cab with peer 4516a714250344268ce0b17c65356895
W20240503 02:11:52.494344 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:52.580889 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:52.637230 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:52.757678 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:52.783339 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:11:52.866658 23310 raft_consensus.cc:1203] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
W20240503 02:11:52.963521 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:53.049585 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:53.156558 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:11:53.317242 24052 raft_consensus.cc:1050] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 4516a714250344268ce0b17c65356895 to VOTER
I20240503 02:11:53.317982 24052 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:11:53.320950 23310 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:11:53.321615 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:11:53.322162 24068 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:11:53.322537 24052 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:11:53.322821 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:53.326831 24052 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.327733 23310 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.332243 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.332489 21390 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 reported cstate change: config changed from index 5 to 6, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.339203 21668 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:11:53.341214 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:11:53.341516 23310 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:11:53.342154 24068 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:53.342880 24068 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:53.345402 24070 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.346349 23310 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.346704 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:11:53.350215 21381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aeb0c27e2f5f4649bce1259462637cab with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:11:53.351905 21396 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 reported cstate change: config changed from index 6 to 7, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
W20240503 02:11:53.358356 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aeb0c27e2f5f4649bce1259462637cab: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:53.402376 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:11:53.607272 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } }
I20240503 02:11:53.609778 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:11:53.610134 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:53.610342 24092 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:53.613523 23958 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
I20240503 02:11:53.614185 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
I20240503 02:11:53.619216 21390 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:53.625205 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:53.628607 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:11:53.629400 23958 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:11:53.633086 23958 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:11:53.633976 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:11:53.638067 21382 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 323dc3d2761e4114a2f87832d2510ef4 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:11:53.640038 21392 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
W20240503 02:11:53.642357 21735 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 02fea8896de14021872010185f9408a7 (127.20.218.69:39821): Couldn't send request to peer 02fea8896de14021872010185f9408a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 323dc3d2761e4114a2f87832d2510ef4. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:53.643194 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:11:53.686247 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:53.700027 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:11:53.754858 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:54.048516 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 02:11:54.080394 24113 ts_tablet_manager.cc:902] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:11:54.082625 24113 tablet_copy_client.cc:308] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:11:54.092171 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 02fea8896de14021872010185f9408a7 ({username='slave'} at 127.20.218.69:44443)
I20240503 02:11:54.092432 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 02fea8896de14021872010185f9408a7 at {username='slave'} at 127.20.218.69:44443: session id = 02fea8896de14021872010185f9408a7-323dc3d2761e4114a2f87832d2510ef4
I20240503 02:11:54.095216 21819 tablet_copy_source_session.cc:205] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:54.100298 24113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:54.111775 24113 tablet_copy_client.cc:783] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:54.112160 24113 tablet_copy_client.cc:647] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:54.114742 24117 tablet_copy_client.cc:1002] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: tablet copy: Injecting latency into file download: 1500
W20240503 02:11:54.230365 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:11:54.428711 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:55.123035 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.189736 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.235491 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.246218 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.251705 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.568107 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240503 02:11:55.569473 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aeb0c27e2f5f4649bce1259462637cab: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:55.615428 24113 tablet_copy_client.cc:523] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:55.619184 24113 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Bootstrap starting.
I20240503 02:11:55.629143 24113 log.cc:824] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:55.664774 24113 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:55.665309 24113 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Bootstrap complete.
I20240503 02:11:55.665805 24113 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Time spent bootstrapping tablet: real 0.047s	user 0.044s	sys 0.004s
I20240503 02:11:55.670356 24113 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:55.670810 24113 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 02fea8896de14021872010185f9408a7, State: Initialized, Role: LEARNER
I20240503 02:11:55.671314 24113 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:55.673005 24113 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 02:11:55.674168 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 02fea8896de14021872010185f9408a7-323dc3d2761e4114a2f87832d2510ef4 received from {username='slave'} at 127.20.218.69:44443
I20240503 02:11:55.674335 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 02fea8896de14021872010185f9408a7-323dc3d2761e4114a2f87832d2510ef4 on tablet 323dc3d2761e4114a2f87832d2510ef4 with peer 02fea8896de14021872010185f9408a7
I20240503 02:11:55.690510 23047 raft_consensus.cc:1203] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6]   Dedup: 2.6->[]
W20240503 02:11:55.789436 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:11:55.963645 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:11:56.226787 24176 raft_consensus.cc:1050] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 02fea8896de14021872010185f9408a7 to VOTER
I20240503 02:11:56.227862 24176 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:11:56.232903 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:11:56.233839 24130 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:11:56.234134 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:56.238044 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:11:56.238978 24176 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:56.242576 24180 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.243403 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.249089 21392 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:56.248852 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.255442 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:11:56.257531 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:56.258165 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:56.258134 24130 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:56.258894 24130 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:56.260562 24180 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.261421 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.262005 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:56.264838 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 323dc3d2761e4114a2f87832d2510ef4 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:11:56.266976 21390 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
W20240503 02:11:56.287058 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:56.457552 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:11:56.600327 21668 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } }
I20240503 02:11:56.603533 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:11:56.603993 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:56.604481 24203 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:56.608211 24070 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
I20240503 02:11:56.609182 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
W20240503 02:11:56.610452 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 323dc3d2761e4114a2f87832d2510ef4: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:11:56.614624 21392 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:56.620645 21668 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:56.626331 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 02:11:56.632591 21604 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 02fea8896de14021872010185f9408a7 (127.20.218.69:39821): Couldn't send request to peer 02fea8896de14021872010185f9408a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e357b085a8e48d9901ac648dd89e31f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:56.637028 24210 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:11:56.638047 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:56.640282 24070 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:11:56.641950 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:11:56.643932 21381 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e357b085a8e48d9901ac648dd89e31f with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:11:56.645648 21396 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:57.039646 24218 ts_tablet_manager.cc:902] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Initiating tablet copy from peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:11:57.041446 24218 tablet_copy_client.cc:308] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34055
I20240503 02:11:57.056592 21688 tablet_copy_service.cc:140] P efb50bbbedb9457f88533cc7e32c793a: Received BeginTabletCopySession request for tablet 7e357b085a8e48d9901ac648dd89e31f from peer 02fea8896de14021872010185f9408a7 ({username='slave'} at 127.20.218.69:35209)
I20240503 02:11:57.056821 21688 tablet_copy_service.cc:161] P efb50bbbedb9457f88533cc7e32c793a: Beginning new tablet copy session on tablet 7e357b085a8e48d9901ac648dd89e31f from peer 02fea8896de14021872010185f9408a7 at {username='slave'} at 127.20.218.69:35209: session id = 02fea8896de14021872010185f9408a7-7e357b085a8e48d9901ac648dd89e31f
I20240503 02:11:57.060137 21688 tablet_copy_source_session.cc:205] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:57.061753 24218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e357b085a8e48d9901ac648dd89e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:57.067189 24218 tablet_copy_client.cc:783] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:57.067610 24218 tablet_copy_client.cc:647] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:57.069918 24222 tablet_copy_client.cc:1002] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: tablet copy: Injecting latency into file download: 1500
W20240503 02:11:57.528419 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:57.632298 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:57.645668 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aeb0c27e2f5f4649bce1259462637cab: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:11:57.671003 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:58.034447 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:11:58.314878 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:11:58.509650 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:11:58.570480 24218 tablet_copy_client.cc:523] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.573452 24218 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Bootstrap starting.
I20240503 02:11:58.608127 24218 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.608573 24218 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Bootstrap complete.
I20240503 02:11:58.608867 24218 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Time spent bootstrapping tablet: real 0.036s	user 0.030s	sys 0.004s
I20240503 02:11:58.609763 24218 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:58.609988 24218 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 02fea8896de14021872010185f9408a7, State: Initialized, Role: LEARNER
I20240503 02:11:58.610183 24218 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:11:58.610920 24218 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 02:11:58.612002 21688 tablet_copy_service.cc:342] P efb50bbbedb9457f88533cc7e32c793a: Request end of tablet copy session 02fea8896de14021872010185f9408a7-7e357b085a8e48d9901ac648dd89e31f received from {username='slave'} at 127.20.218.69:35209
I20240503 02:11:58.612140 21688 tablet_copy_service.cc:433] P efb50bbbedb9457f88533cc7e32c793a: ending tablet copy session 02fea8896de14021872010185f9408a7-7e357b085a8e48d9901ac648dd89e31f on tablet 7e357b085a8e48d9901ac648dd89e31f with peer 02fea8896de14021872010185f9408a7
W20240503 02:11:58.712699 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:11:58.793294 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:11:59.049221 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240503 02:11:59.074709 23047 raft_consensus.cc:1203] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6]   Dedup: 2.6->[]
W20240503 02:11:59.205592 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:11:59.323359 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:11:59.411576 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:11:59.700140 24301 raft_consensus.cc:1050] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 02fea8896de14021872010185f9408a7 to VOTER
I20240503 02:11:59.701009 24301 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:11:59.703903 23047 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:11:59.703929 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:11:59.704500 24300 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:11:59.704830 24303 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:11:59.705138 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:59.707602 24300 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.708348 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.713128 23047 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.713693 21396 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.725634 21667 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:11:59.727610 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:59.727819 23047 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:11:59.728451 24300 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:59.728722 24301 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:11:59.731235 24301 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.732323 23047 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.733719 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:11:59.735718 21381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e357b085a8e48d9901ac648dd89e31f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:11:59.738162 21392 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:11:59.751582 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:00.009418 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:12:00.022346 21799 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } }
I20240503 02:12:00.025833 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240503 02:12:00.026156 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:00.026582 24326 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:00.030086 24179 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
I20240503 02:12:00.030889 21929 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } }
I20240503 02:12:00.035595 21390 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:00.039047 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:12:00.041508 21799 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:00.044847 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.045804 24179 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:12:00.046568 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:00.046568 21735 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 02fea8896de14021872010185f9408a7 (127.20.218.69:39821): Couldn't send request to peer 02fea8896de14021872010185f9408a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a236b927e104bf48a35c975a5507c5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:00.048897 24179 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:12:00.049557 21929 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:12:00.053274 21382 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a236b927e104bf48a35c975a5507c5e with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:12:00.055173 21390 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 5 to 6, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
W20240503 02:12:00.145689 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:12:00.529840 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:12:00.532263 24345 ts_tablet_manager.cc:902] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:00.533807 24345 tablet_copy_client.cc:308] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:00.534641 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 2a236b927e104bf48a35c975a5507c5e from peer 02fea8896de14021872010185f9408a7 ({username='slave'} at 127.20.218.69:44443)
I20240503 02:12:00.534852 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 2a236b927e104bf48a35c975a5507c5e from peer 02fea8896de14021872010185f9408a7 at {username='slave'} at 127.20.218.69:44443: session id = 02fea8896de14021872010185f9408a7-2a236b927e104bf48a35c975a5507c5e
I20240503 02:12:00.537053 21819 tablet_copy_source_session.cc:205] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:00.538380 24345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a236b927e104bf48a35c975a5507c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:00.543310 24345 tablet_copy_client.cc:783] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:00.543596 24345 tablet_copy_client.cc:647] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:00.545295 24349 tablet_copy_client.cc:1002] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:00.887917 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240503 02:12:00.889048 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:00.947748 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:12:01.099510 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:12:01.675448 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:12:01.932554 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:02.045725 24345 tablet_copy_client.cc:523] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:02.048518 24345 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Bootstrap starting.
I20240503 02:12:02.087400 24345 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:02.087949 24345 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Bootstrap complete.
I20240503 02:12:02.088292 24345 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Time spent bootstrapping tablet: real 0.040s	user 0.039s	sys 0.000s
I20240503 02:12:02.089157 24345 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:02.089490 24345 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 02fea8896de14021872010185f9408a7, State: Initialized, Role: LEARNER
I20240503 02:12:02.089699 24345 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:02.090631 24345 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 02:12:02.091588 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 02fea8896de14021872010185f9408a7-2a236b927e104bf48a35c975a5507c5e received from {username='slave'} at 127.20.218.69:44443
I20240503 02:12:02.091732 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 02fea8896de14021872010185f9408a7-2a236b927e104bf48a35c975a5507c5e on tablet 2a236b927e104bf48a35c975a5507c5e with peer 02fea8896de14021872010185f9408a7
I20240503 02:12:02.129544 23047 raft_consensus.cc:1203] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6]   Dedup: 2.6->[]
W20240503 02:12:02.319778 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:12:02.542719 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:12:02.560453 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:12:02.737844 24412 raft_consensus.cc:1050] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 02fea8896de14021872010185f9408a7 to VOTER
I20240503 02:12:02.738543 24412 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:02.741395 21930 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 02:12:02.741684 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:02.741721 23047 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:02.742067 24413 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:02.742303 24415 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:02.745146 24413 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.745908 21930 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.749738 23047 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.750298 21396 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 6 to 7, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:02.757627 21798 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:02.759675 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:12:02.760035 23047 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:12:02.760586 24365 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:02.761045 24365 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:02.763142 24365 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.765985 23047 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.766469 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a236b927e104bf48a35c975a5507c5e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:02.766714 21929 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:02.771924 21392 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 7 to 8, VOTER 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
W20240503 02:12:02.785517 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:12:02.983987 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2a236b927e104bf48a35c975a5507c5e: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:03.131004 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.156203 21909 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:03.166949 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.182796 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.188294 21777 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:12:03.221001 21646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:03.228168 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:12:03.339650 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:12:03.417722 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.439340 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.449362 21909 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:03.454013 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.467483 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:03.488353 21777 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:12:03.505638 21646 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240503 02:12:03.531466 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:12:03.583016 21929 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:03.586539 21668 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:03.587116 21798 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:03.587188 24514 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:03.587731 24513 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:03.591732 24279 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:03.592861 21668 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
W20240503 02:12:03.594307 21865 consensus_peers.cc:485] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305): Couldn't send request to peer 5b7a113589d64600a80987859abf90bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 143591b9348249c7836e2f121eb1b073. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:03.599179 21392 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:03.602499 21798 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:03.620164 21798 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:03.622570 23047 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:03.622896 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:03.623301 24506 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:03.623786 24365 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:03.626526 24506 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:12:03.627323 23047 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
W20240503 02:12:03.629429 21734 consensus_peers.cc:485] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 -> Peer 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117): Couldn't send request to peer 0b5ef04c277e4fb09b5dcb9314371503. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a236b927e104bf48a35c975a5507c5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:03.629494 21929 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
W20240503 02:12:03.633831 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:12:03.634351 21392 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:03.650364 21929 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:03.653316 21798 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:03.653510 21668 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:03.654217 24279 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:03.654896 24514 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:03.659169 24279 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:03.659873 21798 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
W20240503 02:12:03.663259 21864 consensus_peers.cc:485] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 4516a714250344268ce0b17c65356895 (127.20.218.71:44273): Couldn't send request to peer 4516a714250344268ce0b17c65356895. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c11c73db59748b59b8fba963f543fbb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:03.664072 21668 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:03.665215 21396 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:03.962006 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:12:03.963409 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:12:03.988873 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:04.072444 24569 ts_tablet_manager.cc:902] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:04.074155 24569 tablet_copy_client.cc:308] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:04.080363 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 2a236b927e104bf48a35c975a5507c5e from peer 0b5ef04c277e4fb09b5dcb9314371503 ({username='slave'} at 127.20.218.70:48277)
I20240503 02:12:04.080556 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 2a236b927e104bf48a35c975a5507c5e from peer 0b5ef04c277e4fb09b5dcb9314371503 at {username='slave'} at 127.20.218.70:48277: session id = 0b5ef04c277e4fb09b5dcb9314371503-2a236b927e104bf48a35c975a5507c5e
I20240503 02:12:04.082661 21819 tablet_copy_source_session.cc:205] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:04.083977 24569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a236b927e104bf48a35c975a5507c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.088652 24569 tablet_copy_client.cc:783] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:04.089074 24569 tablet_copy_client.cc:647] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:04.091091 24573 tablet_copy_client.cc:1002] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:04.128489 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:12:04.140089 24575 ts_tablet_manager.cc:902] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Initiating tablet copy from peer 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:12:04.141880 24575 tablet_copy_client.cc:308] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:46877
I20240503 02:12:04.172109 24597 ts_tablet_manager.cc:902] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Initiating tablet copy from peer 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:12:04.177958 21950 tablet_copy_service.cc:140] P 4a634cdcef0a4c3ba054106d3c65f9ae: Received BeginTabletCopySession request for tablet 1c11c73db59748b59b8fba963f543fbb from peer 4516a714250344268ce0b17c65356895 ({username='slave'} at 127.20.218.71:48287)
I20240503 02:12:04.178361 21950 tablet_copy_service.cc:161] P 4a634cdcef0a4c3ba054106d3c65f9ae: Beginning new tablet copy session on tablet 1c11c73db59748b59b8fba963f543fbb from peer 4516a714250344268ce0b17c65356895 at {username='slave'} at 127.20.218.71:48287: session id = 4516a714250344268ce0b17c65356895-1c11c73db59748b59b8fba963f543fbb
I20240503 02:12:04.180770 24597 tablet_copy_client.cc:308] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:46877
I20240503 02:12:04.182436 21950 tablet_copy_source_session.cc:205] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:04.184568 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c11c73db59748b59b8fba963f543fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.190059 24575 tablet_copy_client.cc:783] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:04.190153 21949 tablet_copy_service.cc:140] P 4a634cdcef0a4c3ba054106d3c65f9ae: Received BeginTabletCopySession request for tablet 143591b9348249c7836e2f121eb1b073 from peer 5b7a113589d64600a80987859abf90bd ({username='slave'} at 127.20.218.72:58657)
I20240503 02:12:04.190411 21949 tablet_copy_service.cc:161] P 4a634cdcef0a4c3ba054106d3c65f9ae: Beginning new tablet copy session on tablet 143591b9348249c7836e2f121eb1b073 from peer 5b7a113589d64600a80987859abf90bd at {username='slave'} at 127.20.218.72:58657: session id = 5b7a113589d64600a80987859abf90bd-143591b9348249c7836e2f121eb1b073
I20240503 02:12:04.190492 24575 tablet_copy_client.cc:647] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:04.193454 24579 tablet_copy_client.cc:1002] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: tablet copy: Injecting latency into file download: 1500
I20240503 02:12:04.193866 21949 tablet_copy_source_session.cc:205] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:04.200134 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143591b9348249c7836e2f121eb1b073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:04.212808 24597 tablet_copy_client.cc:783] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:04.213219 24597 tablet_copy_client.cc:647] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:04.216063 24601 tablet_copy_client.cc:1002] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:04.425797 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:12:04.998201 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:12:05.012001 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2a236b927e104bf48a35c975a5507c5e: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:05.068416 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:12:05.158444 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:12:05.591526 24569 tablet_copy_client.cc:523] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:05.597824 24569 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap starting.
I20240503 02:12:05.638626 24569 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:05.639093 24569 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap complete.
I20240503 02:12:05.639374 24569 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Time spent bootstrapping tablet: real 0.042s	user 0.027s	sys 0.011s
I20240503 02:12:05.640745 24569 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:05.640975 24569 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Initialized, Role: LEARNER
I20240503 02:12:05.641182 24569 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:05.641918 24569 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:05.642807 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-2a236b927e104bf48a35c975a5507c5e received from {username='slave'} at 127.20.218.70:48277
I20240503 02:12:05.642957 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-2a236b927e104bf48a35c975a5507c5e on tablet 2a236b927e104bf48a35c975a5507c5e with peer 0b5ef04c277e4fb09b5dcb9314371503
I20240503 02:12:05.694027 24575 tablet_copy_client.cc:523] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:05.696904 24575 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Bootstrap starting.
W20240503 02:12:05.700477 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:12:05.716704 24597 tablet_copy_client.cc:523] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:05.721783 24597 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Bootstrap starting.
I20240503 02:12:05.735765 24597 log.cc:824] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:05.738637 24575 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:05.739063 24575 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Bootstrap complete.
I20240503 02:12:05.739326 24575 ts_tablet_manager.cc:1367] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Time spent bootstrapping tablet: real 0.043s	user 0.043s	sys 0.000s
I20240503 02:12:05.740198 24575 raft_consensus.cc:348] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:05.740429 24575 raft_consensus.cc:729] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4516a714250344268ce0b17c65356895, State: Initialized, Role: LEARNER
I20240503 02:12:05.740675 24575 consensus_queue.cc:260] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:05.742842 24575 ts_tablet_manager.cc:1398] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.002s
I20240503 02:12:05.743697 21949 tablet_copy_service.cc:342] P 4a634cdcef0a4c3ba054106d3c65f9ae: Request end of tablet copy session 4516a714250344268ce0b17c65356895-1c11c73db59748b59b8fba963f543fbb received from {username='slave'} at 127.20.218.71:48287
I20240503 02:12:05.743912 21949 tablet_copy_service.cc:433] P 4a634cdcef0a4c3ba054106d3c65f9ae: ending tablet copy session 4516a714250344268ce0b17c65356895-1c11c73db59748b59b8fba963f543fbb on tablet 1c11c73db59748b59b8fba963f543fbb with peer 4516a714250344268ce0b17c65356895
I20240503 02:12:05.773711 24597 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:05.774264 24597 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Bootstrap complete.
I20240503 02:12:05.774718 24597 ts_tablet_manager.cc:1367] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Time spent bootstrapping tablet: real 0.053s	user 0.043s	sys 0.008s
I20240503 02:12:05.779567 24597 raft_consensus.cc:348] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:05.780036 24597 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7a113589d64600a80987859abf90bd, State: Initialized, Role: LEARNER
I20240503 02:12:05.780575 24597 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:05.782781 24597 ts_tablet_manager.cc:1398] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20240503 02:12:05.784161 21949 tablet_copy_service.cc:342] P 4a634cdcef0a4c3ba054106d3c65f9ae: Request end of tablet copy session 5b7a113589d64600a80987859abf90bd-143591b9348249c7836e2f121eb1b073 received from {username='slave'} at 127.20.218.72:58657
I20240503 02:12:05.784327 21949 tablet_copy_service.cc:433] P 4a634cdcef0a4c3ba054106d3c65f9ae: ending tablet copy session 5b7a113589d64600a80987859abf90bd-143591b9348249c7836e2f121eb1b073 on tablet 143591b9348249c7836e2f121eb1b073 with peer 5b7a113589d64600a80987859abf90bd
I20240503 02:12:05.815806 23310 raft_consensus.cc:1203] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 02:12:06.071583 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:12:06.121986 23178 raft_consensus.cc:1203] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
W20240503 02:12:06.135262 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:12:06.339828 23441 raft_consensus.cc:1203] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:12:06.410533 24758 raft_consensus.cc:1050] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: attempting to promote NON_VOTER 4516a714250344268ce0b17c65356895 to VOTER
I20240503 02:12:06.411437 24758 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:06.414086 23310 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.414654 21668 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.414672 24759 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.415169 24758 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.415441 21798 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.417062 24759 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.419096 24759 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.419898 21667 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.421044 21798 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.424814 21392 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.428112 23310 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.447036 21929 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:06.449069 21798 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:06.449113 23310 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:06.449699 24759 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:06.450273 24759 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:06.452091 24619 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.452657 23310 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.453760 21798 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.457051 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c11c73db59748b59b8fba963f543fbb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:06.459048 21393 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 reported cstate change: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:06.463372 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c11c73db59748b59b8fba963f543fbb with delete_type TABLET_DATA_TOMBSTONED (TS efb50bbbedb9457f88533cc7e32c793a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55230
I20240503 02:12:06.464318 24771 tablet_replica.cc:330] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: stopping tablet replica
I20240503 02:12:06.464733 24771 raft_consensus.cc:2229] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:06.465103 24771 raft_consensus.cc:2258] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:06.466543 24771 ts_tablet_manager.cc:1875] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:06.471462 24771 ts_tablet_manager.cc:1888] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:12:06.471657 24771 log.cc:1196] T 1c11c73db59748b59b8fba963f543fbb P efb50bbbedb9457f88533cc7e32c793a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c11c73db59748b59b8fba963f543fbb
I20240503 02:12:06.472628 21381 catalog_manager.cc:4830] TS efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): tablet 1c11c73db59748b59b8fba963f543fbb (table rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]) successfully deleted
I20240503 02:12:06.591784 21667 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
W20240503 02:12:06.594831 21602 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a -> Peer 4516a714250344268ce0b17c65356895 (127.20.218.71:44273): Couldn't send request to peer 4516a714250344268ce0b17c65356895. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b131174c34438d8f72fb2473f47dfa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:06.594852 23178 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:06.594857 21798 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:06.595503 24794 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:06.595868 24301 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:06.599239 24301 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:06.599848 23178 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:06.603844 21798 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:06.606977 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: config changed from index 8 to 9, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:06.612848 24757 raft_consensus.cc:1050] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 to VOTER
I20240503 02:12:06.614385 24757 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:06.618971 23178 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:06.619150 23047 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:06.619446 21929 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:06.619688 24803 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:06.620095 24757 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:06.620434 24621 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:06.624802 24757 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.626665 21929 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.631781 21391 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.635421 23047 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.652715 23178 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.655117 21798 consensus_queue.cc:237] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:06.657280 23047 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:06.657990 24810 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:06.659984 23179 raft_consensus.cc:1261] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:06.660854 24803 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.661497 24809 consensus_queue.cc:1035] T 2a236b927e104bf48a35c975a5507c5e P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:06.661662 23047 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.666702 23178 raft_consensus.cc:2932] T 2a236b927e104bf48a35c975a5507c5e P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.667227 21396 catalog_manager.cc:5484] T 2a236b927e104bf48a35c975a5507c5e P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 10 to 11, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:06.671841 21909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a236b927e104bf48a35c975a5507c5e with delete_type TABLET_DATA_TOMBSTONED (TS 4a634cdcef0a4c3ba054106d3c65f9ae not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:32798
I20240503 02:12:06.673069 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a236b927e104bf48a35c975a5507c5e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:06.675021 24823 tablet_replica.cc:330] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: stopping tablet replica
I20240503 02:12:06.675621 24823 raft_consensus.cc:2229] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:06.676146 24823 raft_consensus.cc:2258] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:06.677788 24823 ts_tablet_manager.cc:1875] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:06.684919 24823 ts_tablet_manager.cc:1888] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:12:06.685119 24823 log.cc:1196] T 2a236b927e104bf48a35c975a5507c5e P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a236b927e104bf48a35c975a5507c5e
I20240503 02:12:06.686250 21382 catalog_manager.cc:4830] TS 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877): tablet 2a236b927e104bf48a35c975a5507c5e (table rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]) successfully deleted
W20240503 02:12:06.747336 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:12:06.770958 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240503 02:12:06.772392 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:06.905382 24619 raft_consensus.cc:1050] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: attempting to promote NON_VOTER 5b7a113589d64600a80987859abf90bd to VOTER
I20240503 02:12:06.906535 24619 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:06.909579 21667 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.909570 21799 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.910228 24758 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.910400 23441 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.911157 24619 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.911643 24758 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.927699 24758 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.929558 21799 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.934281 21667 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.936602 23441 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.939651 21396 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 4 to 5, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.949532 21929 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:06.952292 21667 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:06.952349 23441 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:06.952975 24767 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:06.953383 24758 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:06.957095 24619 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.957968 21667 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.960606 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 143591b9348249c7836e2f121eb1b073 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:06.961798 23441 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:06.962620 21396 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 5 to 6, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:06.966845 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 143591b9348249c7836e2f121eb1b073 with delete_type TABLET_DATA_TOMBSTONED (TS b98f966b15514275b82deb8c7dadcac9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43528
I20240503 02:12:06.977967 24850 tablet_replica.cc:330] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: stopping tablet replica
I20240503 02:12:06.978438 24850 raft_consensus.cc:2229] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:06.978736 24850 raft_consensus.cc:2258] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:06.980531 24850 ts_tablet_manager.cc:1875] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:06.988170 24850 ts_tablet_manager.cc:1888] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:12:06.988494 24850 log.cc:1196] T 143591b9348249c7836e2f121eb1b073 P b98f966b15514275b82deb8c7dadcac9: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/143591b9348249c7836e2f121eb1b073
I20240503 02:12:06.989508 21382 catalog_manager.cc:4830] TS b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): tablet 143591b9348249c7836e2f121eb1b073 (table rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]) successfully deleted
I20240503 02:12:06.997705 21667 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:07.000734 23310 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:07.001704 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:07.001767 24301 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:07.002547 24301 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:07.005224 24301 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:07.010413 23310 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
W20240503 02:12:07.014082 21603 consensus_peers.cc:485] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a -> Peer 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305): Couldn't send request to peer 5b7a113589d64600a80987859abf90bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aeb0c27e2f5f4649bce1259462637cab. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:07.015687 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:07.019136 21396 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 7 to 8, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:07.027168 24867 ts_tablet_manager.cc:902] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Initiating tablet copy from peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:12:07.034003 24867 tablet_copy_client.cc:308] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34055
I20240503 02:12:07.034894 21688 tablet_copy_service.cc:140] P efb50bbbedb9457f88533cc7e32c793a: Received BeginTabletCopySession request for tablet 35b131174c34438d8f72fb2473f47dfa from peer 4516a714250344268ce0b17c65356895 ({username='slave'} at 127.20.218.71:43003)
I20240503 02:12:07.035089 21688 tablet_copy_service.cc:161] P efb50bbbedb9457f88533cc7e32c793a: Beginning new tablet copy session on tablet 35b131174c34438d8f72fb2473f47dfa from peer 4516a714250344268ce0b17c65356895 at {username='slave'} at 127.20.218.71:43003: session id = 4516a714250344268ce0b17c65356895-35b131174c34438d8f72fb2473f47dfa
I20240503 02:12:07.037393 21688 tablet_copy_source_session.cc:205] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:07.038767 24867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:07.045563 24867 tablet_copy_client.cc:783] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:07.045895 24867 tablet_copy_client.cc:647] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:07.048427 24871 tablet_copy_client.cc:1002] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: tablet copy: Injecting latency into file download: 1500
I20240503 02:12:07.385816 24887 ts_tablet_manager.cc:902] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Initiating tablet copy from peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:12:07.387768 24887 tablet_copy_client.cc:308] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34055
I20240503 02:12:07.396054 21688 tablet_copy_service.cc:140] P efb50bbbedb9457f88533cc7e32c793a: Received BeginTabletCopySession request for tablet aeb0c27e2f5f4649bce1259462637cab from peer 5b7a113589d64600a80987859abf90bd ({username='slave'} at 127.20.218.72:55791)
I20240503 02:12:07.396314 21688 tablet_copy_service.cc:161] P efb50bbbedb9457f88533cc7e32c793a: Beginning new tablet copy session on tablet aeb0c27e2f5f4649bce1259462637cab from peer 5b7a113589d64600a80987859abf90bd at {username='slave'} at 127.20.218.72:55791: session id = 5b7a113589d64600a80987859abf90bd-aeb0c27e2f5f4649bce1259462637cab
I20240503 02:12:07.398906 21688 tablet_copy_source_session.cc:205] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:07.401429 24887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb0c27e2f5f4649bce1259462637cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:07.406330 24887 tablet_copy_client.cc:783] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:07.406625 24887 tablet_copy_client.cc:647] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:07.408996 24891 tablet_copy_client.cc:1002] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:07.462934 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:12:07.591799 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:12:07.935285 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:12:08.058343 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:12:08.125406 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:08.446472 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:12:08.549018 24867 tablet_copy_client.cc:523] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:08.551975 24867 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Bootstrap starting.
I20240503 02:12:08.585458 24867 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:08.585973 24867 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Bootstrap complete.
I20240503 02:12:08.586287 24867 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Time spent bootstrapping tablet: real 0.034s	user 0.030s	sys 0.005s
I20240503 02:12:08.587201 24867 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:08.587430 24867 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4516a714250344268ce0b17c65356895, State: Initialized, Role: LEARNER
I20240503 02:12:08.587643 24867 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:08.588743 24867 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:12:08.589682 21688 tablet_copy_service.cc:342] P efb50bbbedb9457f88533cc7e32c793a: Request end of tablet copy session 4516a714250344268ce0b17c65356895-35b131174c34438d8f72fb2473f47dfa received from {username='slave'} at 127.20.218.71:43003
I20240503 02:12:08.589841 21688 tablet_copy_service.cc:433] P efb50bbbedb9457f88533cc7e32c793a: ending tablet copy session 4516a714250344268ce0b17c65356895-35b131174c34438d8f72fb2473f47dfa on tablet 35b131174c34438d8f72fb2473f47dfa with peer 4516a714250344268ce0b17c65356895
W20240503 02:12:08.632078 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:12:08.752785 23310 raft_consensus.cc:1203] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 02:12:08.909510 24887 tablet_copy_client.cc:523] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:08.912247 24887 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Bootstrap starting.
I20240503 02:12:08.949395 24887 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:08.949990 24887 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Bootstrap complete.
I20240503 02:12:08.950271 24887 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Time spent bootstrapping tablet: real 0.038s	user 0.031s	sys 0.008s
I20240503 02:12:08.951280 24887 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:08.951498 24887 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7a113589d64600a80987859abf90bd, State: Initialized, Role: LEARNER
I20240503 02:12:08.951689 24887 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:08.952474 24887 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:08.953282 21688 tablet_copy_service.cc:342] P efb50bbbedb9457f88533cc7e32c793a: Request end of tablet copy session 5b7a113589d64600a80987859abf90bd-aeb0c27e2f5f4649bce1259462637cab received from {username='slave'} at 127.20.218.72:55791
I20240503 02:12:08.953459 21688 tablet_copy_service.cc:433] P efb50bbbedb9457f88533cc7e32c793a: ending tablet copy session 5b7a113589d64600a80987859abf90bd-aeb0c27e2f5f4649bce1259462637cab on tablet aeb0c27e2f5f4649bce1259462637cab with peer 5b7a113589d64600a80987859abf90bd
W20240503 02:12:08.993782 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:12:09.035503 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:12:09.152878 24975 raft_consensus.cc:1050] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 4516a714250344268ce0b17c65356895 to VOTER
I20240503 02:12:09.153678 24975 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:09.156081 23310 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:09.156345 23178 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:09.156661 21799 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:09.156919 24975 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:09.157284 24988 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:09.157713 24986 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:09.164340 24988 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:09.165253 23178 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:09.172302 23310 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:09.173379 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 9 to 10, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:09.178164 21799 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:09.278050 21668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:48646
I20240503 02:12:09.278386 21668 raft_consensus.cc:595] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Received request to transfer leadership
I20240503 02:12:09.286144 23441 raft_consensus.cc:1203] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240503 02:12:09.389770 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:12:09.515725 21668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:48672
I20240503 02:12:09.516000 21668 raft_consensus.cc:595] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Received request to transfer leadership
I20240503 02:12:09.574203 24988 raft_consensus.cc:979] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: : Instructing follower b98f966b15514275b82deb8c7dadcac9 to start an election
I20240503 02:12:09.574414 24988 raft_consensus.cc:1067] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Signalling peer b98f966b15514275b82deb8c7dadcac9 to start an election
I20240503 02:12:09.575277 21799 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
 from {username='slave'} at 127.20.218.66:50535
I20240503 02:12:09.575527 21799 raft_consensus.cc:482] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:09.575675 21799 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:09.577682 21799 raft_consensus.cc:504] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:09.578837 21799 leader_election.cc:288] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 3 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117), 4516a714250344268ce0b17c65356895 (127.20.218.71:44273)
I20240503 02:12:09.579288 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:12:09.579428 23178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "0b5ef04c277e4fb09b5dcb9314371503"
I20240503 02:12:09.579548 21668 raft_consensus.cc:3032] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:12:09.579648 21668 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: efb50bbbedb9457f88533cc7e32c793a, State: Running, Role: LEADER
I20240503 02:12:09.579737 23178 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:09.580025 21668 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:09.580469 21668 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:09.582789 23178 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 3.
I20240503 02:12:09.583277 21668 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 3.
I20240503 02:12:09.583747 21733 leader_election.cc:302] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0b5ef04c277e4fb09b5dcb9314371503, b98f966b15514275b82deb8c7dadcac9, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:12:09.584122 25012 raft_consensus.cc:2781] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:09.584259 25012 raft_consensus.cc:686] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Becoming Leader. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:12:09.584542 25012 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:09.585973 23310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "b98f966b15514275b82deb8c7dadcac9" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "4516a714250344268ce0b17c65356895"
I20240503 02:12:09.586356 23310 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:09.587543 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 reported cstate change: term changed from 2 to 3, leader changed from efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) to b98f966b15514275b82deb8c7dadcac9 (127.20.218.67). New cstate: current_term: 3 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:09.588599 23310 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98f966b15514275b82deb8c7dadcac9 in term 3.
W20240503 02:12:09.676775 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:12:09.886574 25040 raft_consensus.cc:1050] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 5b7a113589d64600a80987859abf90bd to VOTER
I20240503 02:12:09.887238 25040 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:09.889482 23441 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:12:09.889925 23310 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:12:09.890362 21929 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:12:09.890424 24988 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:09.890858 25040 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:09.891688 25042 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:12:09.894613 24988 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.895385 21929 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.895373 23310 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.900126 21396 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 8 to 9, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.901062 23441 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.909457 21668 consensus_queue.cc:237] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:09.911839 23441 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:12:09.912437 25042 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:09.913409 23310 raft_consensus.cc:1261] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:12:09.914377 25042 consensus_queue.cc:1035] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240503 02:12:09.915997 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:12:09.917140 25040 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.917912 23309 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.921309 21381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aeb0c27e2f5f4649bce1259462637cab with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:09.922099 23441 raft_consensus.cc:2932] T aeb0c27e2f5f4649bce1259462637cab P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:09.924062 21393 catalog_manager.cc:5484] T aeb0c27e2f5f4649bce1259462637cab P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 9 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:09.929160 21909 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb0c27e2f5f4649bce1259462637cab with delete_type TABLET_DATA_TOMBSTONED (TS 4a634cdcef0a4c3ba054106d3c65f9ae not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:32798
I20240503 02:12:09.929920 25048 tablet_replica.cc:330] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: stopping tablet replica
I20240503 02:12:09.930148 25048 raft_consensus.cc:2229] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:09.930594 25048 raft_consensus.cc:2258] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:09.931887 25048 ts_tablet_manager.cc:1875] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:09.936549 25048 ts_tablet_manager.cc:1888] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:12:09.936674 25048 log.cc:1196] T aeb0c27e2f5f4649bce1259462637cab P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/aeb0c27e2f5f4649bce1259462637cab
I20240503 02:12:09.937570 21382 catalog_manager.cc:4830] TS 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877): tablet aeb0c27e2f5f4649bce1259462637cab (table rebalance_test_table_2 [id=5771016286524e4d8265e21477cc8104]) successfully deleted
I20240503 02:12:09.974864 21668 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:09.977175 23047 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:09.977249 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240503 02:12:09.977437 21603 consensus_peers.cc:485] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a -> Peer 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305): Couldn't send request to peer 5b7a113589d64600a80987859abf90bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e357b085a8e48d9901ac648dd89e31f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:09.977737 24986 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:09.978121 25040 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:09.981649 25040 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:09.982412 23047 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:09.985626 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:09.987293 21391 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 8 to 9, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:09.991094 23309 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:12:09.992399 25012 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:12:10.012529 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aeb0c27e2f5f4649bce1259462637cab: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:10.037818 21668 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:12:10.038457 25012 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:10.043284 23178 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:12:10.044366 25083 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 02:12:10.406690 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:12:10.459331 21799 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:10.461753 23309 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 02:12:10.461951 23178 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 02:12:10.462453 25082 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:12:10.462792 25012 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:12:10.465440 25012 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:10.466173 23309 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:10.466159 23178 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:10.469311 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35b131174c34438d8f72fb2473f47dfa with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:10.470861 21392 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 10 to 12, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New cstate: current_term: 3 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:10.475351 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b131174c34438d8f72fb2473f47dfa with delete_type TABLET_DATA_TOMBSTONED (TS efb50bbbedb9457f88533cc7e32c793a not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:55230
I20240503 02:12:10.476274 25108 tablet_replica.cc:330] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: stopping tablet replica
I20240503 02:12:10.476480 25108 raft_consensus.cc:2229] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:10.476742 25108 raft_consensus.cc:2258] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:10.478793 25108 ts_tablet_manager.cc:1875] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:10.485622 25108 ts_tablet_manager.cc:1888] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240503 02:12:10.485802 25108 log.cc:1196] T 35b131174c34438d8f72fb2473f47dfa P efb50bbbedb9457f88533cc7e32c793a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/35b131174c34438d8f72fb2473f47dfa
I20240503 02:12:10.486660 21381 catalog_manager.cc:4830] TS efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): tablet 35b131174c34438d8f72fb2473f47dfa (table rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]) successfully deleted
I20240503 02:12:10.488375 25114 ts_tablet_manager.cc:902] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Initiating tablet copy from peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055)
I20240503 02:12:10.490471 25114 tablet_copy_client.cc:308] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34055
I20240503 02:12:10.491501 21688 tablet_copy_service.cc:140] P efb50bbbedb9457f88533cc7e32c793a: Received BeginTabletCopySession request for tablet 7e357b085a8e48d9901ac648dd89e31f from peer 5b7a113589d64600a80987859abf90bd ({username='slave'} at 127.20.218.72:55791)
I20240503 02:12:10.491670 21688 tablet_copy_service.cc:161] P efb50bbbedb9457f88533cc7e32c793a: Beginning new tablet copy session on tablet 7e357b085a8e48d9901ac648dd89e31f from peer 5b7a113589d64600a80987859abf90bd at {username='slave'} at 127.20.218.72:55791: session id = 5b7a113589d64600a80987859abf90bd-7e357b085a8e48d9901ac648dd89e31f
I20240503 02:12:10.492331 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:10.493852 21688 tablet_copy_source_session.cc:205] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:10.495136 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:10.495777 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:10.495766 25083 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:12:10.496423 21734 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117): Couldn't send request to peer 0b5ef04c277e4fb09b5dcb9314371503. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 323dc3d2761e4114a2f87832d2510ef4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:10.496840 25114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e357b085a8e48d9901ac648dd89e31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.496780 25083 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240503 02:12:10.499122 25012 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:12:10.499835 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:12:10.504099 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:12:10.505213 25114 tablet_copy_client.cc:783] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:10.505749 25114 tablet_copy_client.cc:647] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:10.506417 21391 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 8 to 9, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:10.508515 25118 tablet_copy_client.cc:1002] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:10.634495 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:12:10.760975 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:12:10.796119 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:12:10.796489 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:10.978751 25157 ts_tablet_manager.cc:902] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:10.980460 25157 tablet_copy_client.cc:308] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:10.981300 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 0b5ef04c277e4fb09b5dcb9314371503 ({username='slave'} at 127.20.218.70:48277)
I20240503 02:12:10.981523 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 0b5ef04c277e4fb09b5dcb9314371503 at {username='slave'} at 127.20.218.70:48277: session id = 0b5ef04c277e4fb09b5dcb9314371503-323dc3d2761e4114a2f87832d2510ef4
I20240503 02:12:10.983985 21819 tablet_copy_source_session.cc:205] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:10.985472 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:10.989966 25157 tablet_copy_client.cc:783] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:10.990214 25157 tablet_copy_client.cc:647] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:10.991894 25161 tablet_copy_client.cc:1002] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:11.205971 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2a236b927e104bf48a35c975a5507c5e: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:11.452222 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:12.009074 25114 tablet_copy_client.cc:523] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:12.012706 25114 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Bootstrap starting.
I20240503 02:12:12.049492 25114 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:12.049978 25114 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Bootstrap complete.
I20240503 02:12:12.050218 25114 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Time spent bootstrapping tablet: real 0.038s	user 0.033s	sys 0.007s
I20240503 02:12:12.050944 25114 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:12.051154 25114 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7a113589d64600a80987859abf90bd, State: Initialized, Role: LEARNER
I20240503 02:12:12.051314 25114 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:12.052102 25114 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:12:12.053012 21688 tablet_copy_service.cc:342] P efb50bbbedb9457f88533cc7e32c793a: Request end of tablet copy session 5b7a113589d64600a80987859abf90bd-7e357b085a8e48d9901ac648dd89e31f received from {username='slave'} at 127.20.218.72:55791
I20240503 02:12:12.053169 21688 tablet_copy_service.cc:433] P efb50bbbedb9457f88533cc7e32c793a: ending tablet copy session 5b7a113589d64600a80987859abf90bd-7e357b085a8e48d9901ac648dd89e31f on tablet 7e357b085a8e48d9901ac648dd89e31f with peer 5b7a113589d64600a80987859abf90bd
I20240503 02:12:12.474705 23441 raft_consensus.cc:1203] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
W20240503 02:12:12.479580 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:12.492405 25157 tablet_copy_client.cc:523] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:12.495811 25157 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap starting.
I20240503 02:12:12.545477 25157 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:12.545884 25157 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap complete.
I20240503 02:12:12.546159 25157 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Time spent bootstrapping tablet: real 0.050s	user 0.045s	sys 0.007s
I20240503 02:12:12.546806 25157 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:12.547009 25157 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Initialized, Role: LEARNER
I20240503 02:12:12.547206 25157 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:12.548593 25157 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:12:12.549463 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-323dc3d2761e4114a2f87832d2510ef4 received from {username='slave'} at 127.20.218.70:48277
I20240503 02:12:12.549594 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-323dc3d2761e4114a2f87832d2510ef4 on tablet 323dc3d2761e4114a2f87832d2510ef4 with peer 0b5ef04c277e4fb09b5dcb9314371503
W20240503 02:12:12.823140 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:12.837492 23178 raft_consensus.cc:1203] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
W20240503 02:12:12.914474 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 02:12:12.920886 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 323dc3d2761e4114a2f87832d2510ef4: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:13.010931 25267 raft_consensus.cc:1050] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a: attempting to promote NON_VOTER 5b7a113589d64600a80987859abf90bd to VOTER
W20240503 02:12:13.012507 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:12:13.012066 25267 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:13.015805 23441 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 LEARNER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.015805 21799 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.016041 23047 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.016731 25267 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.017107 25268 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.017484 25266 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.033504 25267 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.034760 21799 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.035503 23047 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.039482 23441 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.039963 21393 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 9 to 10, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:13.055387 21668 consensus_queue.cc:237] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:13.058480 23441 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:13.058743 23047 raft_consensus.cc:1261] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer efb50bbbedb9457f88533cc7e32c793a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:13.059482 25145 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:13.060104 25268 consensus_queue.cc:1035] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:13.062757 25267 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P efb50bbbedb9457f88533cc7e32c793a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.063786 23047 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.063917 23441 raft_consensus.cc:2932] T 7e357b085a8e48d9901ac648dd89e31f P 5b7a113589d64600a80987859abf90bd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:13.068392 21381 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e357b085a8e48d9901ac648dd89e31f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:13.069461 21392 catalog_manager.cc:5484] T 7e357b085a8e48d9901ac648dd89e31f P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 10 to 11, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "efb50bbbedb9457f88533cc7e32c793a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
W20240503 02:12:13.073283 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:13.075595 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e357b085a8e48d9901ac648dd89e31f with delete_type TABLET_DATA_TOMBSTONED (TS b98f966b15514275b82deb8c7dadcac9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43528
W20240503 02:12:13.082751 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:12:13.085269 25279 tablet_replica.cc:330] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: stopping tablet replica
I20240503 02:12:13.085542 25279 raft_consensus.cc:2229] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:13.086117 25279 raft_consensus.cc:2258] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:13.087028 25279 ts_tablet_manager.cc:1875] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:13.092581 25279 ts_tablet_manager.cc:1888] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:12:13.092725 25279 log.cc:1196] T 7e357b085a8e48d9901ac648dd89e31f P b98f966b15514275b82deb8c7dadcac9: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/7e357b085a8e48d9901ac648dd89e31f
I20240503 02:12:13.093580 21382 catalog_manager.cc:4830] TS b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): tablet 7e357b085a8e48d9901ac648dd89e31f (table rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]) successfully deleted
I20240503 02:12:13.427368 25295 raft_consensus.cc:1050] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 to VOTER
I20240503 02:12:13.428071 25295 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:13.430876 21668 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.430917 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.431634 25207 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.432111 23178 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:13.432065 25295 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.432757 25298 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:13.437281 25296 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.438004 23178 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.439137 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.443742 21668 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.446264 21393 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 9 to 10, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.452394 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:13.458588 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:13.459353 25296 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:13.462572 23179 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:12:13.463163 25296 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.463582 25298 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:12:13.463805 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.465195 23179 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.470008 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 323dc3d2761e4114a2f87832d2510ef4 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:13.470162 21396 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 10 to 11, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:13.474304 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 323dc3d2761e4114a2f87832d2510ef4 with delete_type TABLET_DATA_TOMBSTONED (TS efb50bbbedb9457f88533cc7e32c793a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55230
I20240503 02:12:13.475395 25309 tablet_replica.cc:330] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: stopping tablet replica
I20240503 02:12:13.475629 25309 raft_consensus.cc:2229] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:13.475952 25309 raft_consensus.cc:2258] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:13.477236 25309 ts_tablet_manager.cc:1875] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:13.483078 25309 ts_tablet_manager.cc:1888] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:12:13.483234 25309 log.cc:1196] T 323dc3d2761e4114a2f87832d2510ef4 P efb50bbbedb9457f88533cc7e32c793a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/323dc3d2761e4114a2f87832d2510ef4
I20240503 02:12:13.484016 21381 catalog_manager.cc:4830] TS efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): tablet 323dc3d2761e4114a2f87832d2510ef4 (table rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]) successfully deleted
W20240503 02:12:13.585685 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:12:13.612107 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:13.627139 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:13.632011 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:13.634253 23159 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:13.634399 21777 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:13.637074 23421 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:13.668816 21909 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:13.755823 21799 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:13.757900 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240503 02:12:13.758169 21733 consensus_peers.cc:485] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 -> Peer 4516a714250344268ce0b17c65356895 (127.20.218.71:44273): Couldn't send request to peer 4516a714250344268ce0b17c65356895. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 323dc3d2761e4114a2f87832d2510ef4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:13.758183 23179 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:12:13.758697 25207 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:12:13.758981 25296 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:12:13.761806 25298 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:13.762567 23179 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:13.763123 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:13.767535 21396 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 11 to 12, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:13.783931 21929 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:13.786897 21668 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:12:13.787134 23441 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:12:13.787539 25359 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:13.789103 25359 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:13.790482 24767 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:12:13.791146 21668 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
W20240503 02:12:13.793452 21866 consensus_peers.cc:485] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 02fea8896de14021872010185f9408a7 (127.20.218.69:39821): Couldn't send request to peer 02fea8896de14021872010185f9408a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 143591b9348249c7836e2f121eb1b073. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:13.798378 21390 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index 6 to 7, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
I20240503 02:12:13.807144 23440 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 02fea8896de14021872010185f9408a7 (127.20.218.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } } }
W20240503 02:12:14.183622 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:12:14.287600 25396 ts_tablet_manager.cc:902] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Initiating tablet copy from peer 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:12:14.289013 25396 tablet_copy_client.cc:308] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:46877
I20240503 02:12:14.295887 21949 tablet_copy_service.cc:140] P 4a634cdcef0a4c3ba054106d3c65f9ae: Received BeginTabletCopySession request for tablet 143591b9348249c7836e2f121eb1b073 from peer 02fea8896de14021872010185f9408a7 ({username='slave'} at 127.20.218.69:36613)
I20240503 02:12:14.296087 21949 tablet_copy_service.cc:161] P 4a634cdcef0a4c3ba054106d3c65f9ae: Beginning new tablet copy session on tablet 143591b9348249c7836e2f121eb1b073 from peer 02fea8896de14021872010185f9408a7 at {username='slave'} at 127.20.218.69:36613: session id = 02fea8896de14021872010185f9408a7-143591b9348249c7836e2f121eb1b073
I20240503 02:12:14.298414 21949 tablet_copy_source_session.cc:205] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:14.299767 25396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143591b9348249c7836e2f121eb1b073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:14.305209 25396 tablet_copy_client.cc:783] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:14.305559 25396 tablet_copy_client.cc:647] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:14.307132 25400 tablet_copy_client.cc:1002] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: tablet copy: Injecting latency into file download: 1500
I20240503 02:12:14.332806 25402 ts_tablet_manager.cc:902] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:14.334311 25402 tablet_copy_client.cc:308] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:14.341125 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 4516a714250344268ce0b17c65356895 ({username='slave'} at 127.20.218.71:48283)
I20240503 02:12:14.341357 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 323dc3d2761e4114a2f87832d2510ef4 from peer 4516a714250344268ce0b17c65356895 at {username='slave'} at 127.20.218.71:48283: session id = 4516a714250344268ce0b17c65356895-323dc3d2761e4114a2f87832d2510ef4
I20240503 02:12:14.343336 21819 tablet_copy_source_session.cc:205] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:14.344695 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323dc3d2761e4114a2f87832d2510ef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:14.349499 25402 tablet_copy_client.cc:783] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:14.349807 25402 tablet_copy_client.cc:647] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:14.351811 25406 tablet_copy_client.cc:1002] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:14.394088 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240503 02:12:14.694221 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:12:14.724090 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:12:14.844848 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:12:15.076951 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:12:15.305420 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:12:15.606603 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:12:15.683677 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:15.807587 25396 tablet_copy_client.cc:523] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:15.810545 25396 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Bootstrap starting.
I20240503 02:12:15.852288 25402 tablet_copy_client.cc:523] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:15.855963 25402 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Bootstrap starting.
I20240503 02:12:15.857414 25396 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:15.857853 25396 tablet_bootstrap.cc:492] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Bootstrap complete.
I20240503 02:12:15.858161 25396 ts_tablet_manager.cc:1367] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Time spent bootstrapping tablet: real 0.048s	user 0.042s	sys 0.006s
I20240503 02:12:15.858999 25396 raft_consensus.cc:348] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:15.859387 25396 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 02fea8896de14021872010185f9408a7, State: Initialized, Role: LEARNER
I20240503 02:12:15.859659 25396 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: true } }
I20240503 02:12:15.860589 25396 ts_tablet_manager.cc:1398] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:15.861534 21949 tablet_copy_service.cc:342] P 4a634cdcef0a4c3ba054106d3c65f9ae: Request end of tablet copy session 02fea8896de14021872010185f9408a7-143591b9348249c7836e2f121eb1b073 received from {username='slave'} at 127.20.218.69:36613
I20240503 02:12:15.861704 21949 tablet_copy_service.cc:433] P 4a634cdcef0a4c3ba054106d3c65f9ae: ending tablet copy session 02fea8896de14021872010185f9408a7-143591b9348249c7836e2f121eb1b073 on tablet 143591b9348249c7836e2f121eb1b073 with peer 02fea8896de14021872010185f9408a7
I20240503 02:12:15.895373 25402 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:15.895812 25402 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Bootstrap complete.
I20240503 02:12:15.896075 25402 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Time spent bootstrapping tablet: real 0.040s	user 0.034s	sys 0.003s
I20240503 02:12:15.896813 25402 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:15.897573 25402 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4516a714250344268ce0b17c65356895, State: Initialized, Role: LEARNER
I20240503 02:12:15.897823 25402 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:15.898634 25402 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.004s
I20240503 02:12:15.899506 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 4516a714250344268ce0b17c65356895-323dc3d2761e4114a2f87832d2510ef4 received from {username='slave'} at 127.20.218.71:48283
I20240503 02:12:15.899665 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 4516a714250344268ce0b17c65356895-323dc3d2761e4114a2f87832d2510ef4 on tablet 323dc3d2761e4114a2f87832d2510ef4 with peer 4516a714250344268ce0b17c65356895
W20240503 02:12:16.272835 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:12:16.312233 23047 raft_consensus.cc:1203] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 02:12:16.333619 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:16.360697 23309 raft_consensus.cc:1203] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
W20240503 02:12:16.700011 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:16.790412 25533 raft_consensus.cc:1050] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 4516a714250344268ce0b17c65356895 to VOTER
I20240503 02:12:16.791484 25533 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:16.794137 23309 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:12:16.794217 23179 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:12:16.794540 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240503 02:12:16.794849 25430 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:12:16.795480 25533 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:12:16.796221 25536 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:12:16.800403 25533 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:16.801131 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:16.802024 23179 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:16.806546 23309 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:16.806742 21390 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 reported cstate change: config changed from index 12 to 13, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:16.893550 25546 raft_consensus.cc:1050] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: attempting to promote NON_VOTER 02fea8896de14021872010185f9408a7 to VOTER
I20240503 02:12:16.894213 25546 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:16.896824 23047 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 LEARNER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:16.896806 23440 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:16.897107 21668 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:16.897523 25547 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:16.898102 25490 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:16.898382 25546 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:16.903805 25490 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:16.904615 23047 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:16.904672 21668 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:16.909364 23440 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:16.910290 21391 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 7 to 8, 02fea8896de14021872010185f9408a7 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:16.965062 21930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "143591b9348249c7836e2f121eb1b073"
dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60618
I20240503 02:12:16.965392 21930 raft_consensus.cc:595] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:16.985800 21798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4"
dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46524
I20240503 02:12:16.986086 21798 raft_consensus.cc:595] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:12:17.207132 21930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "143591b9348249c7836e2f121eb1b073"
dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60634
I20240503 02:12:17.207403 21930 raft_consensus.cc:595] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:17.227326 21798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4"
dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46542
I20240503 02:12:17.227792 21798 raft_consensus.cc:595] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:12:17.244724 25536 raft_consensus.cc:979] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: : Instructing follower 0b5ef04c277e4fb09b5dcb9314371503 to start an election
I20240503 02:12:17.245121 25534 raft_consensus.cc:1067] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Signalling peer 0b5ef04c277e4fb09b5dcb9314371503 to start an election
I20240503 02:12:17.246205 23179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4"
dest_uuid: "0b5ef04c277e4fb09b5dcb9314371503"
 from {username='slave'} at 127.20.218.67:43561
I20240503 02:12:17.246495 23179 raft_consensus.cc:482] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:17.246616 23179 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:17.248698 23179 raft_consensus.cc:504] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:17.251070 23179 leader_election.cc:288] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [CANDIDATE]: Term 3 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 02fea8896de14021872010185f9408a7 (127.20.218.69:39821), 4516a714250344268ce0b17c65356895 (127.20.218.71:44273)
I20240503 02:12:17.251577 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:12:17.251856 21798 raft_consensus.cc:3032] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:12:17.251972 21798 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:12:17.252322 21798 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:17.252786 21798 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:12:17.253288 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:17.255834 21798 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 3.
I20240503 02:12:17.260262 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "02fea8896de14021872010185f9408a7"
I20240503 02:12:17.260555 23047 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:17.262603 23309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323dc3d2761e4114a2f87832d2510ef4" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: true dest_uuid: "4516a714250344268ce0b17c65356895"
I20240503 02:12:17.262897 23309 raft_consensus.cc:3037] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:12:17.264300 23047 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 3.
I20240503 02:12:17.264953 23115 leader_election.cc:302] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 02fea8896de14021872010185f9408a7, 0b5ef04c277e4fb09b5dcb9314371503, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:12:17.265453 25541 raft_consensus.cc:2781] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:12:17.265465 23309 raft_consensus.cc:2454] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 3.
I20240503 02:12:17.266032 25541 raft_consensus.cc:686] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 LEADER]: Becoming Leader. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Running, Role: LEADER
I20240503 02:12:17.266546 25541 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:17.269919 21396 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: term changed from 2 to 3, leader changed from b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) to 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70). New cstate: current_term: 3 leader_uuid: "0b5ef04c277e4fb09b5dcb9314371503" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:17.365881 25490 raft_consensus.cc:979] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: : Instructing follower 5b7a113589d64600a80987859abf90bd to start an election
I20240503 02:12:17.366096 25490 raft_consensus.cc:1067] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Signalling peer 5b7a113589d64600a80987859abf90bd to start an election
I20240503 02:12:17.367161 23440 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "143591b9348249c7836e2f121eb1b073"
dest_uuid: "5b7a113589d64600a80987859abf90bd"
 from {username='slave'} at 127.20.218.68:35801
I20240503 02:12:17.367553 23440 raft_consensus.cc:482] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:17.367725 23440 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:17.370327 23440 raft_consensus.cc:504] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:17.372629 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "5b7a113589d64600a80987859abf90bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "efb50bbbedb9457f88533cc7e32c793a"
I20240503 02:12:17.372694 21930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "5b7a113589d64600a80987859abf90bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae"
I20240503 02:12:17.372936 21668 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:17.372958 21930 raft_consensus.cc:3032] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:17.372992 23440 leader_election.cc:288] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [CANDIDATE]: Term 2 election: Requested vote from peers efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055), 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877), 02fea8896de14021872010185f9408a7 (127.20.218.69:39821)
I20240503 02:12:17.373051 21930 raft_consensus.cc:729] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a634cdcef0a4c3ba054106d3c65f9ae, State: Running, Role: LEADER
I20240503 02:12:17.373450 21930 consensus_queue.cc:260] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:17.374084 21930 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:17.376339 21930 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7a113589d64600a80987859abf90bd in term 2.
I20240503 02:12:17.376530 21668 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7a113589d64600a80987859abf90bd in term 2.
I20240503 02:12:17.377131 23375 leader_election.cc:302] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a634cdcef0a4c3ba054106d3c65f9ae, 5b7a113589d64600a80987859abf90bd, efb50bbbedb9457f88533cc7e32c793a; no voters: 
I20240503 02:12:17.377581 25553 raft_consensus.cc:2781] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:17.378113 25553 raft_consensus.cc:686] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 2 LEADER]: Becoming Leader. State: Replica: 5b7a113589d64600a80987859abf90bd, State: Running, Role: LEADER
I20240503 02:12:17.378662 25553 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:17.380679 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143591b9348249c7836e2f121eb1b073" candidate_uuid: "5b7a113589d64600a80987859abf90bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "02fea8896de14021872010185f9408a7"
I20240503 02:12:17.380954 23047 raft_consensus.cc:3037] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:17.383502 23047 raft_consensus.cc:2454] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7a113589d64600a80987859abf90bd in term 2.
I20240503 02:12:17.383378 21396 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd reported cstate change: term changed from 1 to 2, leader changed from 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) to 5b7a113589d64600a80987859abf90bd (127.20.218.72). New cstate: current_term: 2 leader_uuid: "5b7a113589d64600a80987859abf90bd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:17.448169 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:17.660207 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 3 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:17.660893 25541 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:17.723479 21798 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:17.724200 25541 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:17.730820 23309 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:17.731758 25628 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:17.820624 21930 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Refusing update from remote peer 5b7a113589d64600a80987859abf90bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:17.821491 25553 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:12:17.829607 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:12:17.939205 23047 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer 5b7a113589d64600a80987859abf90bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:17.939819 25553 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:17.945441 21668 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer 5b7a113589d64600a80987859abf90bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:12:17.946264 25643 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:12:18.084131 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:18.346557 23179 consensus_queue.cc:237] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:18.348899 23309 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 02:12:18.349277 23047 raft_consensus.cc:1261] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 3 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240503 02:12:18.349668 25671 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:12:18.350176 25669 consensus_queue.cc:1035] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:12:18.353399 25669 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:18.354147 23047 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 02fea8896de14021872010185f9408a7 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:18.354394 23309 raft_consensus.cc:2932] T 323dc3d2761e4114a2f87832d2510ef4 P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:18.357889 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 323dc3d2761e4114a2f87832d2510ef4 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:18.359112 21391 catalog_manager.cc:5484] T 323dc3d2761e4114a2f87832d2510ef4 P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: config changed from index 13 to 15, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New cstate: current_term: 3 leader_uuid: "0b5ef04c277e4fb09b5dcb9314371503" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.363124 25145 consensus_queue.cc:579] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.365s)
I20240503 02:12:18.369236 23440 consensus_queue.cc:237] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } }
I20240503 02:12:18.372961 23047 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Refusing update from remote peer 5b7a113589d64600a80987859abf90bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:18.373482 21668 raft_consensus.cc:1261] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Refusing update from remote peer 5b7a113589d64600a80987859abf90bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:12:18.373771 25553 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:18.374460 25645 consensus_queue.cc:1035] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:12:18.376997 25645 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:18.378485 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 323dc3d2761e4114a2f87832d2510ef4 with delete_type TABLET_DATA_TOMBSTONED (TS b98f966b15514275b82deb8c7dadcac9 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:43528
I20240503 02:12:18.379571 25680 tablet_replica.cc:330] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: stopping tablet replica
I20240503 02:12:18.379810 25680 raft_consensus.cc:2229] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.380031 25680 raft_consensus.cc:2258] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.380918 25680 ts_tablet_manager.cc:1875] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:18.380869 21668 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P efb50bbbedb9457f88533cc7e32c793a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:18.382092 23047 raft_consensus.cc:2932] T 143591b9348249c7836e2f121eb1b073 P 02fea8896de14021872010185f9408a7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } } }
I20240503 02:12:18.383528 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 143591b9348249c7836e2f121eb1b073 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:18.385283 21396 catalog_manager.cc:5484] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd reported cstate change: config changed from index 8 to 10, VOTER 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68) evicted. New cstate: current_term: 2 leader_uuid: "5b7a113589d64600a80987859abf90bd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fea8896de14021872010185f9408a7" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:18.390652 21909 tablet_service.cc:1514] Processing DeleteTablet for tablet 143591b9348249c7836e2f121eb1b073 with delete_type TABLET_DATA_TOMBSTONED (TS 4a634cdcef0a4c3ba054106d3c65f9ae not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:32798
I20240503 02:12:18.391230 25680 ts_tablet_manager.cc:1888] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240503 02:12:18.391388 25680 log.cc:1196] T 323dc3d2761e4114a2f87832d2510ef4 P b98f966b15514275b82deb8c7dadcac9: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/323dc3d2761e4114a2f87832d2510ef4
I20240503 02:12:18.392316 21382 catalog_manager.cc:4830] TS b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): tablet 323dc3d2761e4114a2f87832d2510ef4 (table rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]) successfully deleted
I20240503 02:12:18.392683 25682 tablet_replica.cc:330] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: stopping tablet replica
I20240503 02:12:18.392915 25682 raft_consensus.cc:2229] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:18.393143 25682 raft_consensus.cc:2258] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:18.394117 25682 ts_tablet_manager.cc:1875] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:18.398696 25682 ts_tablet_manager.cc:1888] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:12:18.398837 25682 log.cc:1196] T 143591b9348249c7836e2f121eb1b073 P 4a634cdcef0a4c3ba054106d3c65f9ae: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/143591b9348249c7836e2f121eb1b073
I20240503 02:12:18.399652 21382 catalog_manager.cc:4830] TS 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877): tablet 143591b9348249c7836e2f121eb1b073 (table rebalance_test_table_3 [id=b229354f3e9240f396d2129d37f4c11e]) successfully deleted
I20240503 02:12:18.401477 25490 consensus_queue.cc:579] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.200s)
W20240503 02:12:18.405419 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:12:18.500423 25534 consensus_queue.cc:579] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.181s)
I20240503 02:12:18.580742 21930 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:18.584398 21798 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:12:18.585106 23309 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:12:18.585048 25684 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:18.586503 25684 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:18.588904 25684 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
W20240503 02:12:18.591668 21865 consensus_peers.cc:485] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70:36117): Couldn't send request to peer 0b5ef04c277e4fb09b5dcb9314371503. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c11c73db59748b59b8fba963f543fbb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:18.593722 21798 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
I20240503 02:12:18.597292 21393 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae reported cstate change: config changed from index 6 to 7, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:18.598256 23309 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } } }
W20240503 02:12:18.900027 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:12:18.949316 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:12:19.108842 25723 ts_tablet_manager.cc:902] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Initiating tablet copy from peer 4a634cdcef0a4c3ba054106d3c65f9ae (127.20.218.68:46877)
I20240503 02:12:19.110293 25723 tablet_copy_client.cc:308] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:46877
I20240503 02:12:19.117408 21949 tablet_copy_service.cc:140] P 4a634cdcef0a4c3ba054106d3c65f9ae: Received BeginTabletCopySession request for tablet 1c11c73db59748b59b8fba963f543fbb from peer 0b5ef04c277e4fb09b5dcb9314371503 ({username='slave'} at 127.20.218.70:40705)
I20240503 02:12:19.117587 21949 tablet_copy_service.cc:161] P 4a634cdcef0a4c3ba054106d3c65f9ae: Beginning new tablet copy session on tablet 1c11c73db59748b59b8fba963f543fbb from peer 0b5ef04c277e4fb09b5dcb9314371503 at {username='slave'} at 127.20.218.70:40705: session id = 0b5ef04c277e4fb09b5dcb9314371503-1c11c73db59748b59b8fba963f543fbb
I20240503 02:12:19.119580 21949 tablet_copy_source_session.cc:205] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:19.120906 25723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c11c73db59748b59b8fba963f543fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:19.125908 25723 tablet_copy_client.cc:783] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:19.126186 25723 tablet_copy_client.cc:647] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:19.127974 25727 tablet_copy_client.cc:1002] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:19.441028 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:12:19.442261 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2a236b927e104bf48a35c975a5507c5e: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:19.567224 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:12:19.831205 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20240503 02:12:19.862051 25743 consensus_queue.cc:579] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.162s)
I20240503 02:12:19.881409 25686 consensus_queue.cc:579] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.087s)
W20240503 02:12:19.883239 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240503 02:12:19.884428 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:12:19.885123 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:12:20.072301 25743 consensus_queue.cc:579] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.220s)
I20240503 02:12:20.181496 25755 consensus_queue.cc:579] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Leader has been unable to successfully communicate with peer 652a1eb67d644b6cac7d2c95ab4ac939 for more than 30 seconds (30.427s)
W20240503 02:12:20.367127 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:20.628517 25723 tablet_copy_client.cc:523] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:20.632073 25723 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap starting.
I20240503 02:12:20.666020 25723 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:20.666925 25723 tablet_bootstrap.cc:492] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Bootstrap complete.
I20240503 02:12:20.667196 25723 ts_tablet_manager.cc:1367] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Time spent bootstrapping tablet: real 0.035s	user 0.031s	sys 0.005s
I20240503 02:12:20.667871 25723 raft_consensus.cc:348] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:20.668081 25723 raft_consensus.cc:729] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Initialized, Role: LEARNER
I20240503 02:12:20.668274 25723 consensus_queue.cc:260] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:20.668764 25723 ts_tablet_manager.cc:1398] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:12:20.669647 21949 tablet_copy_service.cc:342] P 4a634cdcef0a4c3ba054106d3c65f9ae: Request end of tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-1c11c73db59748b59b8fba963f543fbb received from {username='slave'} at 127.20.218.70:40705
I20240503 02:12:20.669818 21949 tablet_copy_service.cc:433] P 4a634cdcef0a4c3ba054106d3c65f9ae: ending tablet copy session 0b5ef04c277e4fb09b5dcb9314371503-1c11c73db59748b59b8fba963f543fbb on tablet 1c11c73db59748b59b8fba963f543fbb with peer 0b5ef04c277e4fb09b5dcb9314371503
W20240503 02:12:20.826956 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:12:20.855300 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:21.045709 23179 raft_consensus.cc:1203] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 02:12:21.621179 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:12:21.646462 25804 raft_consensus.cc:1050] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae: attempting to promote NON_VOTER 0b5ef04c277e4fb09b5dcb9314371503 to VOTER
I20240503 02:12:21.647300 25804 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:21.651253 23309 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:21.651297 21798 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:21.651527 23179 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 LEARNER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:12:21.652024 25806 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:21.652434 25804 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:21.652698 25803 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:12:21.655980 25806 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.656734 23179 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.656787 21798 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.660936 23309 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.661368 21393 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: config changed from index 7 to 8, 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.669557 21930 consensus_queue.cc:237] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } }
I20240503 02:12:21.671437 23309 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:12:21.671898 23179 raft_consensus.cc:1261] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 FOLLOWER]: Refusing update from remote peer 4a634cdcef0a4c3ba054106d3c65f9ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:12:21.672165 25806 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:21.672605 25803 consensus_queue.cc:1035] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:12:21.675022 25804 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.675674 23179 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.678022 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c11c73db59748b59b8fba963f543fbb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:21.678264 23309 raft_consensus.cc:2932] T 1c11c73db59748b59b8fba963f543fbb P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.679929 21390 catalog_manager.cc:5484] T 1c11c73db59748b59b8fba963f543fbb P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: config changed from index 8 to 9, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } }
I20240503 02:12:21.684145 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c11c73db59748b59b8fba963f543fbb with delete_type TABLET_DATA_TOMBSTONED (TS b98f966b15514275b82deb8c7dadcac9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43528
I20240503 02:12:21.684861 25815 tablet_replica.cc:330] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: stopping tablet replica
I20240503 02:12:21.685037 25815 raft_consensus.cc:2229] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:21.685460 25815 raft_consensus.cc:2258] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:21.686374 25815 ts_tablet_manager.cc:1875] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:12:21.687633 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:21.691895 25815 ts_tablet_manager.cc:1888] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:12:21.692070 25815 log.cc:1196] T 1c11c73db59748b59b8fba963f543fbb P b98f966b15514275b82deb8c7dadcac9: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c11c73db59748b59b8fba963f543fbb
I20240503 02:12:21.692880 21382 catalog_manager.cc:4830] TS b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): tablet 1c11c73db59748b59b8fba963f543fbb (table rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]) successfully deleted
I20240503 02:12:21.810249 21798 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:21.813220 23310 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:12:21.814311 25828 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:12:21.815191 23179 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 02:12:21.815768 25828 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 02:12:21.818043 25828 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:21.819013 23310 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:21.823789 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 reported cstate change: config changed from index 12 to 13, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New cstate: current_term: 3 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
I20240503 02:12:21.823750 23179 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 5b7a113589d64600a80987859abf90bd (127.20.218.72) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } } }
W20240503 02:12:21.830590 21734 consensus_peers.cc:485] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 -> Peer 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305): Couldn't send request to peer 5b7a113589d64600a80987859abf90bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35b131174c34438d8f72fb2473f47dfa. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:22.173424 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:22.341964 25849 ts_tablet_manager.cc:902] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:22.343308 25849 tablet_copy_client.cc:308] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:22.349552 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 35b131174c34438d8f72fb2473f47dfa from peer 5b7a113589d64600a80987859abf90bd ({username='slave'} at 127.20.218.72:32887)
I20240503 02:12:22.349737 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 35b131174c34438d8f72fb2473f47dfa from peer 5b7a113589d64600a80987859abf90bd at {username='slave'} at 127.20.218.72:32887: session id = 5b7a113589d64600a80987859abf90bd-35b131174c34438d8f72fb2473f47dfa
I20240503 02:12:22.351760 21819 tablet_copy_source_session.cc:205] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:22.353075 25849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b131174c34438d8f72fb2473f47dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.357834 25849 tablet_copy_client.cc:783] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:22.358088 25849 tablet_copy_client.cc:647] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:22.360042 25853 tablet_copy_client.cc:1002] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:22.453263 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:12:22.606765 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:12:23.176529 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35b131174c34438d8f72fb2473f47dfa: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:23.255780 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:12:23.269104 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:23.860503 25849 tablet_copy_client.cc:523] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:23.863324 25849 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Bootstrap starting.
I20240503 02:12:23.898538 25849 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:23.898955 25849 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Bootstrap complete.
I20240503 02:12:23.899201 25849 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Time spent bootstrapping tablet: real 0.036s	user 0.036s	sys 0.000s
I20240503 02:12:23.900022 25849 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:23.900266 25849 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7a113589d64600a80987859abf90bd, State: Initialized, Role: LEARNER
I20240503 02:12:23.900470 25849 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: NON_VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: true } }
I20240503 02:12:23.901208 25849 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:23.902033 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 5b7a113589d64600a80987859abf90bd-35b131174c34438d8f72fb2473f47dfa received from {username='slave'} at 127.20.218.72:32887
I20240503 02:12:23.902156 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 5b7a113589d64600a80987859abf90bd-35b131174c34438d8f72fb2473f47dfa on tablet 35b131174c34438d8f72fb2473f47dfa with peer 5b7a113589d64600a80987859abf90bd
W20240503 02:12:23.966770 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:12:23.970863 23440 raft_consensus.cc:1203] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 LEARNER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13]   Dedup: 3.13->[]
W20240503 02:12:24.130575 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:12:24.466284 25870 raft_consensus.cc:1050] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 5b7a113589d64600a80987859abf90bd to VOTER
I20240503 02:12:24.467097 25870 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:24.470564 23440 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:24.470587 23179 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:24.470824 23309 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240503 02:12:24.471489 25921 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:24.471954 25870 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:24.472323 25919 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240503 02:12:24.475598 25870 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:24.476500 23440 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:24.477535 23179 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:24.479842 23309 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:24.480873 21396 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 13 to 14, 5b7a113589d64600a80987859abf90bd (127.20.218.72) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.607439 21798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49550
I20240503 02:12:24.607687 21798 raft_consensus.cc:595] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Received request to transfer leadership
W20240503 02:12:24.772300 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:12:24.835801 21798 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49568
I20240503 02:12:24.836038 21798 raft_consensus.cc:595] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Received request to transfer leadership
I20240503 02:12:24.884295 25921 raft_consensus.cc:979] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: : Instructing follower 0b5ef04c277e4fb09b5dcb9314371503 to start an election
I20240503 02:12:24.884508 25921 raft_consensus.cc:1067] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Signalling peer 0b5ef04c277e4fb09b5dcb9314371503 to start an election
I20240503 02:12:24.885267 23179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa"
dest_uuid: "0b5ef04c277e4fb09b5dcb9314371503"
 from {username='slave'} at 127.20.218.67:43561
I20240503 02:12:24.885566 23179 raft_consensus.cc:482] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:24.885677 23179 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 3 FOLLOWER]: Advancing to term 4
W20240503 02:12:24.886977 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:12:24.888090 23179 raft_consensus.cc:504] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
W20240503 02:12:24.888164 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:12:24.888931 23179 leader_election.cc:288] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [CANDIDATE]: Term 4 election: Requested vote from peers b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345), 4516a714250344268ce0b17c65356895 (127.20.218.71:44273), 5b7a113589d64600a80987859abf90bd (127.20.218.72:39305)
I20240503 02:12:24.889791 21798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "b98f966b15514275b82deb8c7dadcac9"
I20240503 02:12:24.890053 21798 raft_consensus.cc:3032] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 02:12:24.890132 21798 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b98f966b15514275b82deb8c7dadcac9, State: Running, Role: LEADER
I20240503 02:12:24.890040 23309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "4516a714250344268ce0b17c65356895"
I20240503 02:12:24.890275 23309 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:24.890466 21798 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:24.890937 21798 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:24.892726 23309 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 4.
I20240503 02:12:24.893925 21798 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 4.
I20240503 02:12:24.894529 23114 leader_election.cc:302] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0b5ef04c277e4fb09b5dcb9314371503, 4516a714250344268ce0b17c65356895, b98f966b15514275b82deb8c7dadcac9; no voters: 
I20240503 02:12:24.894912 25671 raft_consensus.cc:2781] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 02:12:24.896566 25671 raft_consensus.cc:686] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 4 LEADER]: Becoming Leader. State: Replica: 0b5ef04c277e4fb09b5dcb9314371503, State: Running, Role: LEADER
I20240503 02:12:24.896972 23440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b131174c34438d8f72fb2473f47dfa" candidate_uuid: "0b5ef04c277e4fb09b5dcb9314371503" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: true dest_uuid: "5b7a113589d64600a80987859abf90bd"
I20240503 02:12:24.896984 25671 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:24.897290 23440 raft_consensus.cc:3037] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 3 FOLLOWER]: Advancing to term 4
I20240503 02:12:24.899832 23440 raft_consensus.cc:2454] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5ef04c277e4fb09b5dcb9314371503 in term 4.
I20240503 02:12:24.900457 21391 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: term changed from 3 to 4, leader changed from b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) to 0b5ef04c277e4fb09b5dcb9314371503 (127.20.218.70). New cstate: current_term: 4 leader_uuid: "0b5ef04c277e4fb09b5dcb9314371503" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:24.921725 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:12:24.981591 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:12:25.325896 21798 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 4 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 02:12:25.326683 25671 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 02:12:25.377023 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:12:25.455227 23440 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 4 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 02:12:25.455955 25671 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240503 02:12:25.460721 23309 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 4 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 4 index: 15. (index mismatch)
I20240503 02:12:25.462359 25969 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240503 02:12:25.507606 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:12:25.720674 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:12:25.959825 23179 consensus_queue.cc:237] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 4.15, Last appended by leader: 14, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } }
I20240503 02:12:25.961988 23309 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 4 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240503 02:12:25.962034 23440 raft_consensus.cc:1261] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 4 FOLLOWER]: Refusing update from remote peer 0b5ef04c277e4fb09b5dcb9314371503: Log matching property violated. Preceding OpId in replica: term: 4 index: 15. Preceding OpId from leader: term: 4 index: 16. (index mismatch)
I20240503 02:12:25.962651 25671 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:12:25.963019 25969 consensus_queue.cc:1035] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240503 02:12:25.966486 25967 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 [term 4 LEADER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:25.967229 23440 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 5b7a113589d64600a80987859abf90bd [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:25.967523 23309 raft_consensus.cc:2932] T 35b131174c34438d8f72fb2473f47dfa P 4516a714250344268ce0b17c65356895 [term 4 FOLLOWER]: Committing config change with OpId 4.16: config changed from index 14 to 16, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } } }
I20240503 02:12:25.972810 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35b131174c34438d8f72fb2473f47dfa with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:25.972921 21390 catalog_manager.cc:5484] T 35b131174c34438d8f72fb2473f47dfa P 0b5ef04c277e4fb09b5dcb9314371503 reported cstate change: config changed from index 14 to 16, VOTER b98f966b15514275b82deb8c7dadcac9 (127.20.218.67) evicted. New cstate: current_term: 4 leader_uuid: "0b5ef04c277e4fb09b5dcb9314371503" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7a113589d64600a80987859abf90bd" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:25.979064 21777 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b131174c34438d8f72fb2473f47dfa with delete_type TABLET_DATA_TOMBSTONED (TS b98f966b15514275b82deb8c7dadcac9 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:43528
I20240503 02:12:25.979736 25996 tablet_replica.cc:330] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: stopping tablet replica
I20240503 02:12:25.979959 25996 raft_consensus.cc:2229] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:25.980170 25996 raft_consensus.cc:2258] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:25.981647 25996 ts_tablet_manager.cc:1875] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:25.986418 25996 ts_tablet_manager.cc:1888] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.15
I20240503 02:12:25.986579 25996 log.cc:1196] T 35b131174c34438d8f72fb2473f47dfa P b98f966b15514275b82deb8c7dadcac9: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/35b131174c34438d8f72fb2473f47dfa
I20240503 02:12:25.987700 21382 catalog_manager.cc:4830] TS b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): tablet 35b131174c34438d8f72fb2473f47dfa (table rebalance_test_table_4 [id=8b591d01392b4704b546043be13a65c4]) successfully deleted
I20240503 02:12:26.258050 23159 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:26.258713 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:26.269026 21909 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:26.269963 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:26.290228 23421 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:26.352720 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:26.386919 21777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:12:26.424700 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:12:26.426848 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aeb0c27e2f5f4649bce1259462637cab: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
W20240503 02:12:26.435009 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:12:26.446601 21798 consensus_queue.cc:237] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:26.449906 21929 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:12:26.450172 21733 consensus_peers.cc:485] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 -> Peer 4516a714250344268ce0b17c65356895 (127.20.218.71:44273): Couldn't send request to peer 4516a714250344268ce0b17c65356895. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99da2b3ee7cc414e8642cf26d46a3cc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:26.450294 21668 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:26.450618 26052 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:26.451195 25991 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:26.455777 21668 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:26.454996 26052 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:26.459466 21929 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
I20240503 02:12:26.461905 21393 catalog_manager.cc:5484] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a reported cstate change: config changed from index -1 to 4, NON_VOTER 4516a714250344268ce0b17c65356895 (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } } }
W20240503 02:12:26.624776 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:12:26.921604 26072 ts_tablet_manager.cc:902] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Initiating tablet copy from peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345)
I20240503 02:12:26.923128 26072 tablet_copy_client.cc:308] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45345
I20240503 02:12:26.924063 21819 tablet_copy_service.cc:140] P b98f966b15514275b82deb8c7dadcac9: Received BeginTabletCopySession request for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 from peer 4516a714250344268ce0b17c65356895 ({username='slave'} at 127.20.218.71:48283)
I20240503 02:12:26.924247 21819 tablet_copy_service.cc:161] P b98f966b15514275b82deb8c7dadcac9: Beginning new tablet copy session on tablet 99da2b3ee7cc414e8642cf26d46a3cc3 from peer 4516a714250344268ce0b17c65356895 at {username='slave'} at 127.20.218.71:48283: session id = 4516a714250344268ce0b17c65356895-99da2b3ee7cc414e8642cf26d46a3cc3
I20240503 02:12:26.928202 21819 tablet_copy_source_session.cc:205] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:26.929967 26072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99da2b3ee7cc414e8642cf26d46a3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:26.936686 26072 tablet_copy_client.cc:783] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:26.936952 26072 tablet_copy_client.cc:647] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:26.938802 26076 tablet_copy_client.cc:1002] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: tablet copy: Injecting latency into file download: 1500
W20240503 02:12:27.019044 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:12:27.408152 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:12:27.885174 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:12:28.231058 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:12:28.410204 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:12:28.439294 26072 tablet_copy_client.cc:523] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:28.442642 26072 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Bootstrap starting.
I20240503 02:12:28.475878 26072 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:28.476423 26072 tablet_bootstrap.cc:492] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Bootstrap complete.
I20240503 02:12:28.476686 26072 ts_tablet_manager.cc:1367] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Time spent bootstrapping tablet: real 0.034s	user 0.027s	sys 0.006s
I20240503 02:12:28.477476 26072 raft_consensus.cc:348] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:28.477689 26072 raft_consensus.cc:729] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4516a714250344268ce0b17c65356895, State: Initialized, Role: LEARNER
I20240503 02:12:28.477911 26072 consensus_queue.cc:260] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: NON_VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: true } }
I20240503 02:12:28.478673 26072 ts_tablet_manager.cc:1398] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:28.479487 21819 tablet_copy_service.cc:342] P b98f966b15514275b82deb8c7dadcac9: Request end of tablet copy session 4516a714250344268ce0b17c65356895-99da2b3ee7cc414e8642cf26d46a3cc3 received from {username='slave'} at 127.20.218.71:48283
I20240503 02:12:28.479641 21819 tablet_copy_service.cc:433] P b98f966b15514275b82deb8c7dadcac9: ending tablet copy session 4516a714250344268ce0b17c65356895-99da2b3ee7cc414e8642cf26d46a3cc3 on tablet 99da2b3ee7cc414e8642cf26d46a3cc3 with peer 4516a714250344268ce0b17c65356895
I20240503 02:12:28.549419 23310 raft_consensus.cc:1203] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 02:12:28.759330 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:12:28.989890 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:12:29.126459 26138 raft_consensus.cc:1050] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9: attempting to promote NON_VOTER 4516a714250344268ce0b17c65356895 to VOTER
I20240503 02:12:29.128015 26138 consensus_queue.cc:237] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:29.130919 23310 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 LEARNER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:29.130964 21929 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:29.130941 21668 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:29.131552 26141 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:29.132000 26138 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:29.132278 26107 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:29.138692 26138 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.140877 21668 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.143544 23310 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.143770 21929 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.148517 21390 catalog_manager.cc:5484] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 4 to 5, 4516a714250344268ce0b17c65356895 (127.20.218.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:29.166373 21798 consensus_queue.cc:237] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } }
I20240503 02:12:29.168881 21929 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:29.169090 23310 raft_consensus.cc:1261] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Refusing update from remote peer b98f966b15514275b82deb8c7dadcac9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:29.169557 26107 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:29.169991 26139 consensus_queue.cc:1035] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:29.172748 26138 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.173705 23310 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4516a714250344268ce0b17c65356895 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.173799 21929 raft_consensus.cc:2932] T 99da2b3ee7cc414e8642cf26d46a3cc3 P 4a634cdcef0a4c3ba054106d3c65f9ae [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } } }
I20240503 02:12:29.179069 21382 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:29.179466 21393 catalog_manager.cc:5484] T 99da2b3ee7cc414e8642cf26d46a3cc3 P b98f966b15514275b82deb8c7dadcac9 reported cstate change: config changed from index 5 to 6, VOTER efb50bbbedb9457f88533cc7e32c793a (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "b98f966b15514275b82deb8c7dadcac9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4516a714250344268ce0b17c65356895" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 44273 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:29.183693 21646 tablet_service.cc:1514] Processing DeleteTablet for tablet 99da2b3ee7cc414e8642cf26d46a3cc3 with delete_type TABLET_DATA_TOMBSTONED (TS efb50bbbedb9457f88533cc7e32c793a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55230
I20240503 02:12:29.184660 26151 tablet_replica.cc:330] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: stopping tablet replica
I20240503 02:12:29.184906 26151 raft_consensus.cc:2229] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:29.185287 26151 raft_consensus.cc:2258] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:29.186409 26151 ts_tablet_manager.cc:1875] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:29.192044 26151 ts_tablet_manager.cc:1888] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:12:29.192320 26151 log.cc:1196] T 99da2b3ee7cc414e8642cf26d46a3cc3 P efb50bbbedb9457f88533cc7e32c793a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/99da2b3ee7cc414e8642cf26d46a3cc3
I20240503 02:12:29.193224 21381 catalog_manager.cc:4830] TS efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): tablet 99da2b3ee7cc414e8642cf26d46a3cc3 (table rebalance_test_table_1 [id=d00137cf5c2a46cba10d527ebf4b98f1]) successfully deleted
I20240503 02:12:29.350960 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:29.370213 23159 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:29.370973 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:29.390561 23421 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:29.394691 21909 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:29.417060 21777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:12:29.421658 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:12:29.422219 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 323dc3d2761e4114a2f87832d2510ef4: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:29.466033 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:29.633517 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:29.635982 23159 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:29.645466 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:29.679212 23421 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:29.680337 21909 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:29.700744 21777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:29.747959 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:12:29.823173 21734 consensus_peers.cc:485] T 5eceeb73c14743d88e84b9a69d3129d7 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:12:29.980114 21734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:12:30.019282 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:39543
--local_ip_for_outbound_sockets=127.20.218.65
--tserver_master_addrs=127.20.218.126:41289
--webserver_port=40195
--webserver_interface=127.20.218.65
--builtin_ntp_servers=127.20.218.84:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240503 02:12:30.123188 21603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:12:30.154659 21865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:12:30.394484 26231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:30.394845 26231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:30.395018 26231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:30.403383 26231 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:30.403558 26231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:30.403735 26231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:12:30.413895 26231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:39543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=40195
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.20.218.126:41289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:30.415354 26231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:30.417603 26231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:30.431381 26237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:30.431380 26239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:30.431425 26236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:30.432754 26231 server_base.cc:1034] running on GCE node
I20240503 02:12:30.433539 26231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:30.434541 26231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:30.435786 26231 hybrid_clock.cc:648] HybridClock initialized: now 1714702350435712 us; error 71 us; skew 500 ppm
I20240503 02:12:30.436262 26231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:30.446858 26231 webserver.cc:460] Webserver started at http://127.20.218.65:40195/ using document root <none> and password file <none>
I20240503 02:12:30.447746 26231 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:30.447885 26231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:30.454963 26231 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240503 02:12:30.460387 26245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:30.462205 26231 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:12:30.462422 26231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "652a1eb67d644b6cac7d2c95ab4ac939"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-81lq"
I20240503 02:12:30.463418 26231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:30.495360 26231 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:30.496292 26231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:30.496608 26231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:30.497612 26231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:30.499989 26253 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240503 02:12:30.531601 26231 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240503 02:12:30.531740 26231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.032s	user 0.002s	sys 0.000s
I20240503 02:12:30.531859 26231 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240503 02:12:30.535897 26253 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
W20240503 02:12:30.550637 21603 consensus_peers.cc:485] T 2cae4071aaec4fa4992fb330f51d85fc P efb50bbbedb9457f88533cc7e32c793a -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:12:30.559001 26253 log.cc:824] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:30.562002 26231 ts_tablet_manager.cc:590] Registered 12 tablets
I20240503 02:12:30.562148 26231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.030s	user 0.029s	sys 0.000s
I20240503 02:12:30.614475 26231 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:39543
I20240503 02:12:30.614523 26360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:39543 every 8 connection(s)
I20240503 02:12:30.615993 26253 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.616238 26231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:30.616601 26253 tablet_bootstrap.cc:492] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.618247 26253 ts_tablet_manager.cc:1367] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.083s	user 0.061s	sys 0.019s
I20240503 02:12:30.623139 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26231
I20240503 02:12:30.625986 26253 raft_consensus.cc:348] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:12:30.627163 26253 raft_consensus.cc:729] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.627861 26253 consensus_queue.cc:260] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } }
I20240503 02:12:30.629875 26253 ts_tablet_manager.cc:1398] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.004s
I20240503 02:12:30.630378 26253 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.633488 26361 heartbeater.cc:347] Connected to a master server at 127.20.218.126:41289
I20240503 02:12:30.634227 26361 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:30.635555 26361 heartbeater.cc:511] Master 127.20.218.126:41289 requested a full tablet report, sending...
I20240503 02:12:30.639036 21393 ts_manager.cc:194] Re-registered known tserver with Master: 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543)
I20240503 02:12:30.642586 21393 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:52917
I20240503 02:12:30.645864 26361 heartbeater.cc:503] Master 127.20.218.126:41289 was elected leader, sending a full tablet report...
W20240503 02:12:30.661640 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240503 02:12:30.698643 26253 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.699308 26253 tablet_bootstrap.cc:492] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.700582 26253 ts_tablet_manager.cc:1367] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.070s	user 0.039s	sys 0.027s
I20240503 02:12:30.701678 26253 raft_consensus.cc:348] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:12:30.701898 26253 raft_consensus.cc:729] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.702234 26253 consensus_queue.cc:260] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:12:30.702785 26253 ts_tablet_manager.cc:1398] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:30.703078 26253 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.753070 26253 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.753602 26253 tablet_bootstrap.cc:492] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.754626 26253 ts_tablet_manager.cc:1367] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.052s	user 0.034s	sys 0.015s
I20240503 02:12:30.755348 26253 raft_consensus.cc:348] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.755559 26253 raft_consensus.cc:729] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.755826 26253 consensus_queue.cc:260] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.756350 26253 ts_tablet_manager.cc:1398] T 323dc3d2761e4114a2f87832d2510ef4 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:30.756673 26253 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.799569 26253 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.800091 26253 tablet_bootstrap.cc:492] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.801306 26253 ts_tablet_manager.cc:1367] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.045s	user 0.029s	sys 0.011s
I20240503 02:12:30.802062 26253 raft_consensus.cc:348] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.802263 26253 raft_consensus.cc:729] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.802520 26253 consensus_queue.cc:260] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.802991 26253 ts_tablet_manager.cc:1398] T 2cae4071aaec4fa4992fb330f51d85fc P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:30.803267 26253 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.831670 26316 raft_consensus.cc:3037] T 5eceeb73c14743d88e84b9a69d3129d7 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:30.852972 26253 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.853618 26253 tablet_bootstrap.cc:492] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.854636 26253 ts_tablet_manager.cc:1367] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.051s	user 0.030s	sys 0.019s
I20240503 02:12:30.855389 26253 raft_consensus.cc:348] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:30.855630 26253 raft_consensus.cc:729] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.855917 26253 consensus_queue.cc:260] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } attrs { replace: true } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "0b5ef04c277e4fb09b5dcb9314371503" member_type: NON_VOTER last_known_addr { host: "127.20.218.70" port: 36117 } attrs { promote: true } }
I20240503 02:12:30.856495 26253 ts_tablet_manager.cc:1398] T 35b131174c34438d8f72fb2473f47dfa P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:12:30.856747 26253 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.895932 26253 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.896447 26253 tablet_bootstrap.cc:492] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.897456 26253 ts_tablet_manager.cc:1367] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.041s	user 0.029s	sys 0.011s
I20240503 02:12:30.898164 26253 raft_consensus.cc:348] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:12:30.898344 26253 raft_consensus.cc:729] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.898645 26253 consensus_queue.cc:260] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:12:30.899147 26253 ts_tablet_manager.cc:1398] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:30.899420 26253 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.942704 26253 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.943197 26253 tablet_bootstrap.cc:492] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.944234 26253 ts_tablet_manager.cc:1367] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.045s	user 0.032s	sys 0.009s
I20240503 02:12:30.944907 26253 raft_consensus.cc:348] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.945123 26253 raft_consensus.cc:729] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:30.945436 26253 consensus_queue.cc:260] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:30.945960 26253 ts_tablet_manager.cc:1398] T 9e03e6f7855b48b5a35fcd84a9279714 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.003s
I20240503 02:12:30.946378 26253 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:30.958107 26316 raft_consensus.cc:3037] T 33e2bb56fb72428cbbebfa69ba154306 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:30.998361 26253 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:30.998909 26253 tablet_bootstrap.cc:492] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:30.999905 26253 ts_tablet_manager.cc:1367] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.054s	user 0.042s	sys 0.008s
I20240503 02:12:31.000684 26253 raft_consensus.cc:348] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.000927 26253 raft_consensus.cc:729] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.001245 26253 consensus_queue.cc:260] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.001833 26253 ts_tablet_manager.cc:1398] T daa192225bab430484e3e822a63d6316 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:31.002138 26253 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:31.043350 26253 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:31.043958 26253 tablet_bootstrap.cc:492] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:31.045058 26253 ts_tablet_manager.cc:1367] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.043s	user 0.027s	sys 0.011s
I20240503 02:12:31.045814 26253 raft_consensus.cc:348] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.046029 26253 raft_consensus.cc:729] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.046289 26253 consensus_queue.cc:260] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.046825 26253 ts_tablet_manager.cc:1398] T 2a236b927e104bf48a35c975a5507c5e P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:12:31.047076 26253 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:31.092797 26253 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:31.093303 26253 tablet_bootstrap.cc:492] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:31.094364 26253 ts_tablet_manager.cc:1367] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.047s	user 0.024s	sys 0.020s
I20240503 02:12:31.095108 26253 raft_consensus.cc:348] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:12:31.095302 26253 raft_consensus.cc:729] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.095602 26253 consensus_queue.cc:260] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } }
I20240503 02:12:31.096130 26253 ts_tablet_manager.cc:1398] T aeb0c27e2f5f4649bce1259462637cab P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:12:31.096506 26253 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:31.120537 26316 raft_consensus.cc:3037] T 63989b9f652141379ec1ae9628f73300 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:31.152280 26253 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:31.152797 26253 tablet_bootstrap.cc:492] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:31.154150 26253 ts_tablet_manager.cc:1367] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.058s	user 0.044s	sys 0.010s
I20240503 02:12:31.155174 26253 raft_consensus.cc:348] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:12:31.155459 26253 raft_consensus.cc:729] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.155804 26253 consensus_queue.cc:260] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "4a634cdcef0a4c3ba054106d3c65f9ae" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 46877 } }
I20240503 02:12:31.156456 26253 ts_tablet_manager.cc:1398] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:31.156785 26253 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap starting.
I20240503 02:12:31.211726 26253 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:31.212327 26253 tablet_bootstrap.cc:492] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Bootstrap complete.
I20240503 02:12:31.213620 26253 ts_tablet_manager.cc:1367] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent bootstrapping tablet: real 0.057s	user 0.046s	sys 0.009s
I20240503 02:12:31.214605 26253 raft_consensus.cc:348] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.214870 26253 raft_consensus.cc:729] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a1eb67d644b6cac7d2c95ab4ac939, State: Initialized, Role: FOLLOWER
I20240503 02:12:31.215219 26253 consensus_queue.cc:260] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b98f966b15514275b82deb8c7dadcac9" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45345 } } peers { permanent_uuid: "efb50bbbedb9457f88533cc7e32c793a" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34055 } } peers { permanent_uuid: "652a1eb67d644b6cac7d2c95ab4ac939" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 39543 } }
I20240503 02:12:31.215770 26253 ts_tablet_manager.cc:1398] T 7e357b085a8e48d9901ac648dd89e31f P 652a1eb67d644b6cac7d2c95ab4ac939: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:12:31.285209 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.289666 23159 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:31.310420 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.319002 21909 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:31.340240 21777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:31.377130 23421 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:31.390594 26296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.398676 26316 raft_consensus.cc:3037] T c047b5ecea1a4e2da5c2322d5eed0373 P 652a1eb67d644b6cac7d2c95ab4ac939 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:31.483008 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:31.679998 23027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.688597 23290 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.692034 23159 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:31.724798 26296 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:31.727979 23421 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:31.729372 21909 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:31.748356 21777 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:31.770395 21646 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:31.963883 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26231
I20240503 02:12:31.971071 26355 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:32.084059 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26231
W20240503 02:12:32.108300 21865 connection.cc:535] client connection to 127.20.218.65:39543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:32.108410 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21586
W20240503 02:12:32.111107 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:32.112157 21734 consensus_peers.cc:485] T daa192225bab430484e3e822a63d6316 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.116183 21708 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:32.172809 21865 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:32.304060 21734 consensus_peers.cc:485] T 63989b9f652141379ec1ae9628f73300 P b98f966b15514275b82deb8c7dadcac9 -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.315150 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21586
W20240503 02:12:32.348079 21864 connection.cc:535] client connection to 127.20.218.66:34055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:12:32.348721 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21717
W20240503 02:12:32.348935 23375 connection.cc:535] client connection to 127.20.218.66:34055 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:32.349187 23375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:12:32.349741 21864 consensus_peers.cc:485] T 33e2bb56fb72428cbbebfa69ba154306 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): Couldn't send request to peer efb50bbbedb9457f88533cc7e32c793a. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:32.351668 23375 consensus_peers.cc:485] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd -> Peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): Couldn't send request to peer efb50bbbedb9457f88533cc7e32c793a. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.355701 21839 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:32.387485 21865 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): Couldn't send request to peer 652a1eb67d644b6cac7d2c95ab4ac939. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:12:32.431543 21864 consensus_peers.cc:485] T c047b5ecea1a4e2da5c2322d5eed0373 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer efb50bbbedb9457f88533cc7e32c793a (127.20.218.66:34055): Couldn't send request to peer efb50bbbedb9457f88533cc7e32c793a. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.551206 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21717
I20240503 02:12:32.587898 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21848
W20240503 02:12:32.588655 21865 consensus_peers.cc:485] T 9e03e6f7855b48b5a35fcd84a9279714 P 4a634cdcef0a4c3ba054106d3c65f9ae -> Peer b98f966b15514275b82deb8c7dadcac9 (127.20.218.67:45345): Couldn't send request to peer b98f966b15514275b82deb8c7dadcac9. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:45345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:32.593811 21969 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:12:32.725420 21382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 02:12:32.726845 21382 catalog_manager.cc:4573] TS 652a1eb67d644b6cac7d2c95ab4ac939 (127.20.218.65:39543): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7e357b085a8e48d9901ac648dd89e31f: Network error: Client connection negotiation failed: client connection to 127.20.218.65:39543: connect: Connection refused (error 111)
I20240503 02:12:32.773984 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21848
I20240503 02:12:32.807986 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 22965
I20240503 02:12:32.814427 23086 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:32.954967 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 22965
I20240503 02:12:32.980691 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23096
I20240503 02:12:32.987443 23218 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:33.131209 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23096
W20240503 02:12:33.141048 23377 consensus_peers.cc:485] T 143591b9348249c7836e2f121eb1b073 P 5b7a113589d64600a80987859abf90bd -> Peer 02fea8896de14021872010185f9408a7 (127.20.218.69:39821): Couldn't send request to peer 02fea8896de14021872010185f9408a7. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.69:39821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:33.158603 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23228
I20240503 02:12:33.165683 23350 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:33.301698 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23228
I20240503 02:12:33.326905 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23359
I20240503 02:12:33.334496 23481 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:33.469955 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23359
I20240503 02:12:33.494307 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21365
I20240503 02:12:33.500735 21425 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:33.678189 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21365
2024-05-03T02:12:33Z chronyd exiting
[       OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (70348 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (70348 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240503 02:12:33.753842 21353 test_util.cc:268] Using random seed: 1085335241
I20240503 02:12:33.755645 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:12:33.755743 21353 ts_itest-base.cc:113] --------------
I20240503 02:12:33.755802 21353 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:12:33.755877 21353 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:12:33.755934 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:33Z Disabled control of system clock
I20240503 02:12:33.782784 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:39827
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:39827
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:12:34.128916 26515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:34.129278 26515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:34.129431 26515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:34.137089 26515 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:34.137290 26515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:34.137403 26515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:34.137492 26515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:34.137567 26515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:34.147037 26515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:39827
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:34.148620 26515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:34.150560 26515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:34.162003 26523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:34.162029 26521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:34.162003 26520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:34.162966 26515 server_base.cc:1034] running on GCE node
I20240503 02:12:34.163792 26515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:34.165307 26515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:34.166596 26515 hybrid_clock.cc:648] HybridClock initialized: now 1714702354166537 us; error 48 us; skew 500 ppm
I20240503 02:12:34.166986 26515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:34.169586 26515 webserver.cc:460] Webserver started at http://127.20.218.126:33081/ using document root <none> and password file <none>
I20240503 02:12:34.170369 26515 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:34.170481 26515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:34.170804 26515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:34.173518 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a7e004f2f0f34170837dd7e34e904f70"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.174328 26515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a7e004f2f0f34170837dd7e34e904f70"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.180135 26515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:12:34.183996 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:34.185768 26515 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:12:34.186002 26515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a7e004f2f0f34170837dd7e34e904f70"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.186235 26515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:34.213369 26515 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:34.214434 26515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:34.214792 26515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:34.240099 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:39827 every 8 connection(s)
I20240503 02:12:34.240131 26515 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:39827
I20240503 02:12:34.241886 26515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:12:34.242287 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26515
I20240503 02:12:34.242522 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:12:34.247067 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.258316 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70: Bootstrap starting.
I20240503 02:12:34.261616 26581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.262771 26581 log.cc:824] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:34.265558 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70: No bootstrap required, opened a new log
I20240503 02:12:34.272545 26581 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } }
I20240503 02:12:34.272966 26581 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.273095 26581 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e004f2f0f34170837dd7e34e904f70, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.273788 26581 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } }
I20240503 02:12:34.274004 26581 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:34.274138 26581 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:34.274300 26581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.276479 26581 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } }
I20240503 02:12:34.277048 26581 leader_election.cc:302] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e004f2f0f34170837dd7e34e904f70; no voters: 
I20240503 02:12:34.277735 26581 leader_election.cc:288] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:12:34.277890 26586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.278609 26586 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [term 1 LEADER]: Becoming Leader. State: Replica: a7e004f2f0f34170837dd7e34e904f70, State: Running, Role: LEADER
I20240503 02:12:34.279181 26586 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } }
I20240503 02:12:34.279875 26581 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:34.282662 26588 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e004f2f0f34170837dd7e34e904f70. Latest consensus state: current_term: 1 leader_uuid: "a7e004f2f0f34170837dd7e34e904f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } } }
I20240503 02:12:34.282972 26588 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:34.283113 26587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e004f2f0f34170837dd7e34e904f70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e004f2f0f34170837dd7e34e904f70" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 39827 } } }
I20240503 02:12:34.283507 26587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:34.286700 26595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:34.292117 26595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:34.300700 26595 catalog_manager.cc:1347] Generated new cluster ID: 012d829c7974440d92253eaa2bfe33b3
I20240503 02:12:34.300848 26595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:34.325912 26595 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:34.327289 26595 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:34.335834 26595 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a7e004f2f0f34170837dd7e34e904f70: Generated new TSK 0
I20240503 02:12:34.336767 26595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:34.353209 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:34.697304 26605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:34.697659 26605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:34.697783 26605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:34.705657 26605 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:34.705806 26605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:34.705976 26605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:12:34.715540 26605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:34.716980 26605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:34.718860 26605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:34.731572 26613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:34.731550 26610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:34.731582 26611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:34.732393 26605 server_base.cc:1034] running on GCE node
I20240503 02:12:34.733150 26605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:34.734236 26605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:34.735467 26605 hybrid_clock.cc:648] HybridClock initialized: now 1714702354735427 us; error 45 us; skew 500 ppm
I20240503 02:12:34.735816 26605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:34.738360 26605 webserver.cc:460] Webserver started at http://127.20.218.65:37615/ using document root <none> and password file <none>
I20240503 02:12:34.739145 26605 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:34.739274 26605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:34.739606 26605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:34.744565 26605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6091d2cdbf5b48f7914965f9d8017ecd"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.745473 26605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6091d2cdbf5b48f7914965f9d8017ecd"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.751019 26605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.002s
I20240503 02:12:34.755074 26619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:34.756862 26605 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.004s
I20240503 02:12:34.757069 26605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6091d2cdbf5b48f7914965f9d8017ecd"
format_stamp: "Formatted at 2024-05-03 02:12:34 on dist-test-slave-81lq"
I20240503 02:12:34.757289 26605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:34.772783 26605 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:34.773725 26605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:34.774016 26605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:34.775012 26605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:34.776631 26605 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:34.776742 26605 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:34.776860 26605 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:34.776933 26605 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:34.811461 26605 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:36659
I20240503 02:12:34.811508 26731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:36659 every 8 connection(s)
I20240503 02:12:34.813272 26605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:34.815021 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26605
I20240503 02:12:34.815241 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:12:34.818970 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:34.830384 26732 heartbeater.cc:347] Connected to a master server at 127.20.218.126:39827
I20240503 02:12:34.830818 26732 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:34.831988 26732 heartbeater.cc:511] Master 127.20.218.126:39827 requested a full tablet report, sending...
I20240503 02:12:34.835057 26546 ts_manager.cc:194] Registered new tserver with Master: 6091d2cdbf5b48f7914965f9d8017ecd (127.20.218.65:36659)
I20240503 02:12:34.837697 26546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:48367
W20240503 02:12:35.162493 26736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:35.162809 26736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:35.162932 26736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:35.170593 26736 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:35.170733 26736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:35.170884 26736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:12:35.180533 26736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:35.181984 26736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:35.184516 26736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:35.196794 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:35.196893 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:35.196874 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:35.197901 26736 server_base.cc:1034] running on GCE node
I20240503 02:12:35.198580 26736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:35.199594 26736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:35.200817 26736 hybrid_clock.cc:648] HybridClock initialized: now 1714702355200759 us; error 58 us; skew 500 ppm
I20240503 02:12:35.201160 26736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:35.203799 26736 webserver.cc:460] Webserver started at http://127.20.218.66:44097/ using document root <none> and password file <none>
I20240503 02:12:35.204578 26736 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:35.204702 26736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:35.205024 26736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:35.207780 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ca4c8319c14d433e853e8774372c415b"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.208650 26736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ca4c8319c14d433e853e8774372c415b"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.214712 26736 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20240503 02:12:35.219025 26750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.220927 26736 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:12:35.221125 26736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ca4c8319c14d433e853e8774372c415b"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.221376 26736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:35.236277 26736 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:35.237213 26736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:35.237540 26736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:35.238540 26736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:35.240167 26736 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:35.240283 26736 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.240399 26736 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:35.240470 26736 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.275490 26736 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:37623
I20240503 02:12:35.275542 26862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:37623 every 8 connection(s)
I20240503 02:12:35.276861 26736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:12:35.280288 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26736
I20240503 02:12:35.280503 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:12:35.283944 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:35.291550 26863 heartbeater.cc:347] Connected to a master server at 127.20.218.126:39827
I20240503 02:12:35.291980 26863 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:35.292987 26863 heartbeater.cc:511] Master 127.20.218.126:39827 requested a full tablet report, sending...
I20240503 02:12:35.295208 26546 ts_manager.cc:194] Registered new tserver with Master: ca4c8319c14d433e853e8774372c415b (127.20.218.66:37623)
I20240503 02:12:35.296326 26546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:35859
W20240503 02:12:35.636237 26866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:35.636565 26866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:35.636655 26866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:35.644387 26866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:35.644523 26866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:35.644692 26866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:12:35.654095 26866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:35.655503 26866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:35.657378 26866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:35.669644 26874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:35.669715 26872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:35.669687 26871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:35.670727 26866 server_base.cc:1034] running on GCE node
I20240503 02:12:35.671491 26866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:35.672722 26866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:35.673971 26866 hybrid_clock.cc:648] HybridClock initialized: now 1714702355673917 us; error 36 us; skew 500 ppm
I20240503 02:12:35.674319 26866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:35.683290 26866 webserver.cc:460] Webserver started at http://127.20.218.67:38725/ using document root <none> and password file <none>
I20240503 02:12:35.684382 26866 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:35.684531 26866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:35.684878 26866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:35.688042 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b119d7060d084253b70ce4913cfabc62"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.688889 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b119d7060d084253b70ce4913cfabc62"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.694954 26866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20240503 02:12:35.698578 26880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.699985 26866 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:12:35.700167 26866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b119d7060d084253b70ce4913cfabc62"
format_stamp: "Formatted at 2024-05-03 02:12:35 on dist-test-slave-81lq"
I20240503 02:12:35.700390 26866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:35.719177 26866 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:35.720468 26866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:35.720827 26866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:35.722293 26866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:35.724099 26866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:35.724232 26866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.724356 26866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:35.724448 26866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:35.764160 26866 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:40163
I20240503 02:12:35.764199 26992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:40163 every 8 connection(s)
I20240503 02:12:35.766328 26866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:12:35.776373 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26866
I20240503 02:12:35.776562 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:12:35.780180 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:35.781713 26993 heartbeater.cc:347] Connected to a master server at 127.20.218.126:39827
I20240503 02:12:35.782045 26993 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:35.782902 26993 heartbeater.cc:511] Master 127.20.218.126:39827 requested a full tablet report, sending...
I20240503 02:12:35.784546 26546 ts_manager.cc:194] Registered new tserver with Master: b119d7060d084253b70ce4913cfabc62 (127.20.218.67:40163)
I20240503 02:12:35.785492 26546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:47699
I20240503 02:12:35.841730 26732 heartbeater.cc:503] Master 127.20.218.126:39827 was elected leader, sending a full tablet report...
W20240503 02:12:36.125911 26996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:36.126257 26996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:36.126415 26996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:36.134454 26996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:36.134629 26996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:36.134796 26996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:12:36.144253 26996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:36.145741 26996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:36.147593 26996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:36.160842 27001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:36.160933 27004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:36.160997 27002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:36.162081 26996 server_base.cc:1034] running on GCE node
I20240503 02:12:36.163344 26996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:36.164682 26996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:36.165973 26996 hybrid_clock.cc:648] HybridClock initialized: now 1714702356165916 us; error 48 us; skew 500 ppm
I20240503 02:12:36.166390 26996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:36.169041 26996 webserver.cc:460] Webserver started at http://127.20.218.68:42423/ using document root <none> and password file <none>
I20240503 02:12:36.169929 26996 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:36.170050 26996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:36.170406 26996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:36.173185 26996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1ffccf601c20409cb0a7f5e5a7071838"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.174119 26996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1ffccf601c20409cb0a7f5e5a7071838"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.179800 26996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240503 02:12:36.183669 27010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.185209 26996 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:36.185456 26996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1ffccf601c20409cb0a7f5e5a7071838"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.185712 26996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:36.212936 26996 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:36.213891 26996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:36.214190 26996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:36.215185 26996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:36.216910 26996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:36.217007 26996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.217135 26996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:36.217222 26996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.290457 26996 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:45577
I20240503 02:12:36.290515 27122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:45577 every 8 connection(s)
I20240503 02:12:36.292009 26996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:36.292328 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 26996
I20240503 02:12:36.292496 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:12:36.296185 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:0
--local_ip_for_outbound_sockets=127.20.218.69
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--builtin_ntp_servers=127.20.218.84:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:36.299999 26863 heartbeater.cc:503] Master 127.20.218.126:39827 was elected leader, sending a full tablet report...
I20240503 02:12:36.306798 27123 heartbeater.cc:347] Connected to a master server at 127.20.218.126:39827
I20240503 02:12:36.307123 27123 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:36.307933 27123 heartbeater.cc:511] Master 127.20.218.126:39827 requested a full tablet report, sending...
I20240503 02:12:36.309584 26546 ts_manager.cc:194] Registered new tserver with Master: 1ffccf601c20409cb0a7f5e5a7071838 (127.20.218.68:45577)
I20240503 02:12:36.310930 26546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:35739
W20240503 02:12:36.639438 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:36.639775 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:36.639871 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:36.647440 27127 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:36.647594 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:36.647711 27127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:12:36.657075 27127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:39827
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:36.658457 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:36.660192 27127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:36.672998 27132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:36.673017 27135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:36.673043 27133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:36.673997 27127 server_base.cc:1034] running on GCE node
I20240503 02:12:36.674661 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:36.675676 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:36.676915 27127 hybrid_clock.cc:648] HybridClock initialized: now 1714702356676867 us; error 41 us; skew 500 ppm
I20240503 02:12:36.677280 27127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:36.679973 27127 webserver.cc:460] Webserver started at http://127.20.218.69:37663/ using document root <none> and password file <none>
I20240503 02:12:36.680775 27127 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:36.680883 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:36.681211 27127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:36.684144 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d933092eabe24708a63ddbabb09486a1"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.685007 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d933092eabe24708a63ddbabb09486a1"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.691359 27127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.001s
I20240503 02:12:36.695523 27141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.697247 27127 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:12:36.697520 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d933092eabe24708a63ddbabb09486a1"
format_stamp: "Formatted at 2024-05-03 02:12:36 on dist-test-slave-81lq"
I20240503 02:12:36.697783 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:36.715935 27127 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:36.716866 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:36.717151 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:36.718206 27127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:36.719735 27127 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:36.719826 27127 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.719944 27127 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:36.720028 27127 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:36.754387 27127 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:34429
I20240503 02:12:36.754449 27253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:34429 every 8 connection(s)
I20240503 02:12:36.755796 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:36.756908 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27127
I20240503 02:12:36.757097 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:12:36.771548 27254 heartbeater.cc:347] Connected to a master server at 127.20.218.126:39827
I20240503 02:12:36.771885 27254 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:36.772830 27254 heartbeater.cc:511] Master 127.20.218.126:39827 requested a full tablet report, sending...
I20240503 02:12:36.774554 26545 ts_manager.cc:194] Registered new tserver with Master: d933092eabe24708a63ddbabb09486a1 (127.20.218.69:34429)
I20240503 02:12:36.775490 26545 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:40279
I20240503 02:12:36.788017 21353 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:12:36.788867 26993 heartbeater.cc:503] Master 127.20.218.126:39827 was elected leader, sending a full tablet report...
I20240503 02:12:36.806403 26545 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43152:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:12:36.839936 26798 tablet_service.cc:1467] Processing CreateTablet for tablet 70b57176948c482cae5d4cc0b5d8abc5 (DEFAULT_TABLE table=TestTable [id=80749f948aa2466984670815b7bc9efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.839835 26928 tablet_service.cc:1467] Processing CreateTablet for tablet 70b57176948c482cae5d4cc0b5d8abc5 (DEFAULT_TABLE table=TestTable [id=80749f948aa2466984670815b7bc9efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.840412 27189 tablet_service.cc:1467] Processing CreateTablet for tablet 70b57176948c482cae5d4cc0b5d8abc5 (DEFAULT_TABLE table=TestTable [id=80749f948aa2466984670815b7bc9efe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.841596 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b57176948c482cae5d4cc0b5d8abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.841596 26928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b57176948c482cae5d4cc0b5d8abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.841835 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b57176948c482cae5d4cc0b5d8abc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.854399 27272 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Bootstrap starting.
I20240503 02:12:36.858062 27272 tablet_bootstrap.cc:654] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.858166 27273 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Bootstrap starting.
I20240503 02:12:36.860268 27274 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Bootstrap starting.
I20240503 02:12:36.861284 27272 log.cc:824] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:36.861991 27273 tablet_bootstrap.cc:654] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.863270 27273 log.cc:824] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:36.864486 27274 tablet_bootstrap.cc:654] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.864835 27272 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: No bootstrap required, opened a new log
I20240503 02:12:36.865155 27272 ts_tablet_manager.cc:1367] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240503 02:12:36.865634 27274 log.cc:824] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:36.866413 27273 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: No bootstrap required, opened a new log
I20240503 02:12:36.866726 27273 ts_tablet_manager.cc:1367] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.001s
I20240503 02:12:36.868454 27274 tablet_bootstrap.cc:492] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: No bootstrap required, opened a new log
I20240503 02:12:36.868750 27274 ts_tablet_manager.cc:1367] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240503 02:12:36.875106 27274 raft_consensus.cc:348] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.875556 27274 raft_consensus.cc:374] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.875659 27274 raft_consensus.cc:729] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b119d7060d084253b70ce4913cfabc62, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.875797 27272 raft_consensus.cc:348] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.875903 27273 raft_consensus.cc:348] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.876294 27272 raft_consensus.cc:374] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.876294 27273 raft_consensus.cc:374] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.876405 27273 raft_consensus.cc:729] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4c8319c14d433e853e8774372c415b, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.876405 27272 raft_consensus.cc:729] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d933092eabe24708a63ddbabb09486a1, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.876444 27274 consensus_queue.cc:260] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.877070 27273 consensus_queue.cc:260] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.877070 27272 consensus_queue.cc:260] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.879092 27254 heartbeater.cc:503] Master 127.20.218.126:39827 was elected leader, sending a full tablet report...
I20240503 02:12:36.879179 27273 ts_tablet_manager.cc:1398] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.003s
I20240503 02:12:36.879463 27272 ts_tablet_manager.cc:1398] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.004s
I20240503 02:12:36.882107 27274 ts_tablet_manager.cc:1398] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Time spent starting tablet: real 0.013s	user 0.014s	sys 0.000s
I20240503 02:12:36.948796 27279 raft_consensus.cc:482] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.949074 27279 raft_consensus.cc:504] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.950798 27279 leader_election.cc:288] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b119d7060d084253b70ce4913cfabc62 (127.20.218.67:40163), d933092eabe24708a63ddbabb09486a1 (127.20.218.69:34429)
I20240503 02:12:36.958405 26948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b57176948c482cae5d4cc0b5d8abc5" candidate_uuid: "ca4c8319c14d433e853e8774372c415b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b119d7060d084253b70ce4913cfabc62" is_pre_election: true
I20240503 02:12:36.958508 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b57176948c482cae5d4cc0b5d8abc5" candidate_uuid: "ca4c8319c14d433e853e8774372c415b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d933092eabe24708a63ddbabb09486a1" is_pre_election: true
I20240503 02:12:36.958902 26948 raft_consensus.cc:2454] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4c8319c14d433e853e8774372c415b in term 0.
I20240503 02:12:36.958959 27209 raft_consensus.cc:2454] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4c8319c14d433e853e8774372c415b in term 0.
I20240503 02:12:36.959547 26753 leader_election.cc:302] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b119d7060d084253b70ce4913cfabc62, ca4c8319c14d433e853e8774372c415b; no voters: 
I20240503 02:12:36.959939 27279 raft_consensus.cc:2781] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.960101 27279 raft_consensus.cc:482] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.960213 27279 raft_consensus.cc:3037] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.962327 27279 raft_consensus.cc:504] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.962966 27279 leader_election.cc:288] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [CANDIDATE]: Term 1 election: Requested vote from peers b119d7060d084253b70ce4913cfabc62 (127.20.218.67:40163), d933092eabe24708a63ddbabb09486a1 (127.20.218.69:34429)
I20240503 02:12:36.963568 26948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b57176948c482cae5d4cc0b5d8abc5" candidate_uuid: "ca4c8319c14d433e853e8774372c415b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b119d7060d084253b70ce4913cfabc62"
I20240503 02:12:36.963766 26948 raft_consensus.cc:3037] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.964036 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b57176948c482cae5d4cc0b5d8abc5" candidate_uuid: "ca4c8319c14d433e853e8774372c415b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d933092eabe24708a63ddbabb09486a1"
I20240503 02:12:36.964226 27209 raft_consensus.cc:3037] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.965996 26948 raft_consensus.cc:2454] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4c8319c14d433e853e8774372c415b in term 1.
I20240503 02:12:36.966400 27209 raft_consensus.cc:2454] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4c8319c14d433e853e8774372c415b in term 1.
I20240503 02:12:36.966470 26753 leader_election.cc:302] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b119d7060d084253b70ce4913cfabc62, ca4c8319c14d433e853e8774372c415b; no voters: 
I20240503 02:12:36.966837 27279 raft_consensus.cc:2781] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.967335 27279 raft_consensus.cc:686] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [term 1 LEADER]: Becoming Leader. State: Replica: ca4c8319c14d433e853e8774372c415b, State: Running, Role: LEADER
I20240503 02:12:36.967900 27279 consensus_queue.cc:237] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } }
I20240503 02:12:36.972407 26546 catalog_manager.cc:5484] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b reported cstate change: term changed from 0 to 1, leader changed from <none> to ca4c8319c14d433e853e8774372c415b (127.20.218.66). New cstate: current_term: 1 leader_uuid: "ca4c8319c14d433e853e8774372c415b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 } } peers { permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 } } peers { permanent_uuid: "ca4c8319c14d433e853e8774372c415b" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 37623 } } }
I20240503 02:12:36.994935 21353 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:12:36.997869 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6091d2cdbf5b48f7914965f9d8017ecd to finish bootstrapping
W20240503 02:12:37.008021 27255 tablet.cc:2369] T 70b57176948c482cae5d4cc0b5d8abc5 P d933092eabe24708a63ddbabb09486a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:37.008881 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ca4c8319c14d433e853e8774372c415b to finish bootstrapping
I20240503 02:12:37.017310 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b119d7060d084253b70ce4913cfabc62 to finish bootstrapping
W20240503 02:12:37.019101 26994 tablet.cc:2369] T 70b57176948c482cae5d4cc0b5d8abc5 P b119d7060d084253b70ce4913cfabc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:37.025807 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1ffccf601c20409cb0a7f5e5a7071838 to finish bootstrapping
W20240503 02:12:37.029887 26864 tablet.cc:2369] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:37.035540 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d933092eabe24708a63ddbabb09486a1 to finish bootstrapping
I20240503 02:12:37.313848 27123 heartbeater.cc:503] Master 127.20.218.126:39827 was elected leader, sending a full tablet report...
I20240503 02:12:37.529764 27058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.535362 26667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.537257 26928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.552065 27189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.554060 26798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:37.587313 27320 consensus_queue.cc:1035] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d933092eabe24708a63ddbabb09486a1" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.588204 27320 consensus_queue.cc:1035] T 70b57176948c482cae5d4cc0b5d8abc5 P ca4c8319c14d433e853e8774372c415b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b119d7060d084253b70ce4913cfabc62" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.738375 27058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.748147 26667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.751423 26928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:37.751829 26798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:37.756502 27189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.441383 26667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.443140 26928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.447991 27058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.458434 26798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:38.464856 27189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.624498 27058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.634594 26928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.650523 27189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.651685 26798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:38.653311 26667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.844055 27058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.852392 26928 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.869513 27189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.870038 26667 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:38.870558 26798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:39.028198 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26605
I20240503 02:12:39.029987 26726 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.121001 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26605
I20240503 02:12:39.139405 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26736
I20240503 02:12:39.140455 26857 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.238422 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26736
I20240503 02:12:39.262954 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26866
I20240503 02:12:39.263921 26988 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.361817 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26866
I20240503 02:12:39.380996 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26996
I20240503 02:12:39.382099 27118 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.474023 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26996
I20240503 02:12:39.491964 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27127
I20240503 02:12:39.493023 27248 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.588582 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27127
I20240503 02:12:39.606829 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 26515
I20240503 02:12:39.608007 26576 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:39.722148 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 26515
2024-05-03T02:12:39Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (6013 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6013 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/2
I20240503 02:12:39.767364 21353 test_util.cc:268] Using random seed: 1091348777
I20240503 02:12:39.769454 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:12:39.769532 21353 ts_itest-base.cc:113] --------------
I20240503 02:12:39.769578 21353 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:12:39.769613 21353 ts_itest-base.cc:115] 2 replicas per TS
I20240503 02:12:39.769654 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:39Z Disabled control of system clock
I20240503 02:12:39.798099 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:37537
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:37537
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:12:40.143903 27484 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:12:40.144327 27484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:40.144440 27484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:40.144560 27484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:40.152436 27484 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:40.152601 27484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:40.152719 27484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:40.152791 27484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:40.152882 27484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:40.163015 27484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:37537
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:40.164491 27484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:40.166409 27484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:40.177798 27492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.177846 27490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.177763 27489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:40.178644 27484 server_base.cc:1034] running on GCE node
I20240503 02:12:40.179523 27484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:40.181114 27484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:40.182374 27484 hybrid_clock.cc:648] HybridClock initialized: now 1714702360182297 us; error 62 us; skew 500 ppm
I20240503 02:12:40.182740 27484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:40.185285 27484 webserver.cc:460] Webserver started at http://127.20.218.126:33659/ using document root <none> and password file <none>
I20240503 02:12:40.186232 27484 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:40.186369 27484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:40.186657 27484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:40.189630 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e729927c47f74970b0a958b67ee29059"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.190449 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e729927c47f74970b0a958b67ee29059"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.196115 27484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240503 02:12:40.199985 27498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.201573 27484 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:12:40.201773 27484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e729927c47f74970b0a958b67ee29059"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.201977 27484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:40.223814 27484 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:40.224764 27484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:40.225080 27484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:40.249279 27484 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:37537
I20240503 02:12:40.249305 27549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:37537 every 8 connection(s)
I20240503 02:12:40.250800 27484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:12:40.254796 27550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.259435 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27484
I20240503 02:12:40.259652 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:12:40.265295 27550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059: Bootstrap starting.
I20240503 02:12:40.269246 27550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.270740 27550 log.cc:824] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:40.273912 27550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059: No bootstrap required, opened a new log
I20240503 02:12:40.280368 27550 raft_consensus.cc:348] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } }
I20240503 02:12:40.280786 27550 raft_consensus.cc:374] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.280900 27550 raft_consensus.cc:729] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e729927c47f74970b0a958b67ee29059, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.281635 27550 consensus_queue.cc:260] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } }
I20240503 02:12:40.281868 27550 raft_consensus.cc:388] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:40.282012 27550 raft_consensus.cc:482] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:40.282181 27550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:40.284360 27550 raft_consensus.cc:504] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } }
I20240503 02:12:40.284899 27550 leader_election.cc:302] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e729927c47f74970b0a958b67ee29059; no voters: 
I20240503 02:12:40.285712 27550 leader_election.cc:288] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:12:40.285890 27555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:40.286521 27555 raft_consensus.cc:686] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [term 1 LEADER]: Becoming Leader. State: Replica: e729927c47f74970b0a958b67ee29059, State: Running, Role: LEADER
I20240503 02:12:40.287146 27555 consensus_queue.cc:237] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } }
I20240503 02:12:40.287838 27550 sys_catalog.cc:564] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:40.290318 27557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e729927c47f74970b0a958b67ee29059. Latest consensus state: current_term: 1 leader_uuid: "e729927c47f74970b0a958b67ee29059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } } }
I20240503 02:12:40.290292 27556 sys_catalog.cc:455] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e729927c47f74970b0a958b67ee29059" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e729927c47f74970b0a958b67ee29059" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 37537 } } }
I20240503 02:12:40.290886 27556 sys_catalog.cc:458] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:40.290892 27557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:40.295405 27563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:40.301075 27563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:40.311177 27563 catalog_manager.cc:1347] Generated new cluster ID: a5e2f2e946f74610bae905a37ae60f33
I20240503 02:12:40.311332 27563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:40.324044 27563 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:40.325969 27563 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:40.344230 27563 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e729927c47f74970b0a958b67ee29059: Generated new TSK 0
I20240503 02:12:40.345497 27563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:40.351605 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:40.699012 27574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:40.699337 27574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:40.699429 27574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:40.707340 27574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:40.707487 27574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:40.707608 27574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:12:40.717016 27574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:40.718427 27574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:40.720177 27574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:40.732312 27580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.732367 27582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:40.732389 27579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:40.733232 27574 server_base.cc:1034] running on GCE node
I20240503 02:12:40.733918 27574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:40.734910 27574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:40.736146 27574 hybrid_clock.cc:648] HybridClock initialized: now 1714702360736101 us; error 40 us; skew 500 ppm
I20240503 02:12:40.736529 27574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:40.743525 27574 webserver.cc:460] Webserver started at http://127.20.218.65:34213/ using document root <none> and password file <none>
I20240503 02:12:40.744385 27574 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:40.744522 27574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:40.744848 27574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:40.747649 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.748512 27574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.754199 27574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240503 02:12:40.758213 27588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.759891 27574 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.003s
I20240503 02:12:40.760115 27574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
format_stamp: "Formatted at 2024-05-03 02:12:40 on dist-test-slave-81lq"
I20240503 02:12:40.760358 27574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:40.780664 27574 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:40.781653 27574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:40.781968 27574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:40.782958 27574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:40.784571 27574 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:40.784667 27574 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.784754 27574 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:40.784809 27574 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:40.818245 27574 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:44115
I20240503 02:12:40.818291 27700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:44115 every 8 connection(s)
I20240503 02:12:40.819588 27574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:40.823376 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27574
I20240503 02:12:40.823681 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:12:40.827978 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:40.835095 27701 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:40.835484 27701 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:40.836525 27701 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:40.839047 27515 ts_manager.cc:194] Registered new tserver with Master: ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:40.840646 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:48869
W20240503 02:12:41.176890 27705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:41.177233 27705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:41.177407 27705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:41.185065 27705 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:41.185215 27705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:41.185410 27705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:12:41.195097 27705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:41.196632 27705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:41.198611 27705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:41.211016 27713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:41.211009 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:41.211005 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:41.212406 27705 server_base.cc:1034] running on GCE node
I20240503 02:12:41.213131 27705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:41.214213 27705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:41.215440 27705 hybrid_clock.cc:648] HybridClock initialized: now 1714702361215391 us; error 51 us; skew 500 ppm
I20240503 02:12:41.215782 27705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:41.218382 27705 webserver.cc:460] Webserver started at http://127.20.218.66:46869/ using document root <none> and password file <none>
I20240503 02:12:41.219218 27705 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:41.219355 27705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:41.219681 27705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:41.222479 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "acd475a9eee143a89c1d28a05fa51d26"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.223351 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "acd475a9eee143a89c1d28a05fa51d26"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.228971 27705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240503 02:12:41.232781 27719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.234428 27705 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:41.234628 27705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "acd475a9eee143a89c1d28a05fa51d26"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.234851 27705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:41.268397 27705 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:41.269487 27705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:41.269794 27705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:41.270990 27705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:41.272960 27705 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:41.273064 27705 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.273145 27705 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:41.273204 27705 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.308609 27705 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:46503
I20240503 02:12:41.308755 27831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:46503 every 8 connection(s)
I20240503 02:12:41.310145 27705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:12:41.311053 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27705
I20240503 02:12:41.311234 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:12:41.315282 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:41.327311 27832 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:41.327657 27832 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:41.328466 27832 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:41.329950 27515 ts_manager.cc:194] Registered new tserver with Master: acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:41.330763 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:42205
W20240503 02:12:41.670815 27835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:41.671149 27835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:41.671280 27835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:41.679157 27835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:41.679309 27835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:41.679474 27835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:12:41.688922 27835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:41.690424 27835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:41.692250 27835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:41.704520 27841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:41.704537 27843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:41.704563 27840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:41.705492 27835 server_base.cc:1034] running on GCE node
I20240503 02:12:41.706174 27835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:41.707244 27835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:41.708484 27835 hybrid_clock.cc:648] HybridClock initialized: now 1714702361708425 us; error 50 us; skew 500 ppm
I20240503 02:12:41.708840 27835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:41.711374 27835 webserver.cc:460] Webserver started at http://127.20.218.67:43613/ using document root <none> and password file <none>
I20240503 02:12:41.712154 27835 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:41.712273 27835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:41.712600 27835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:41.715422 27835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eaffa047cf9b43b79a5b75089ea63901"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.716305 27835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eaffa047cf9b43b79a5b75089ea63901"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.722028 27835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240503 02:12:41.725955 27849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.727576 27835 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:12:41.727777 27835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eaffa047cf9b43b79a5b75089ea63901"
format_stamp: "Formatted at 2024-05-03 02:12:41 on dist-test-slave-81lq"
I20240503 02:12:41.727998 27835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:41.752352 27835 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:41.753266 27835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:41.753670 27835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:41.754709 27835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:41.756304 27835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:41.756402 27835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.756490 27835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:41.756544 27835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:41.789752 27835 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:45743
I20240503 02:12:41.789810 27961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:45743 every 8 connection(s)
I20240503 02:12:41.791098 27835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:12:41.797751 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27835
I20240503 02:12:41.797950 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:12:41.800981 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:41.804913 27962 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:41.805261 27962 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:41.806095 27962 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:41.807843 27515 ts_manager.cc:194] Registered new tserver with Master: eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:41.809058 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:60989
I20240503 02:12:41.845180 27701 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
W20240503 02:12:42.153000 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:42.153355 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:42.153492 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:42.161659 27965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:42.161801 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:42.161952 27965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:12:42.171626 27965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:42.173032 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:42.174903 27965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:42.187700 27971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:42.187966 27973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:42.188166 27970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:42.188755 27965 server_base.cc:1034] running on GCE node
I20240503 02:12:42.189430 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:42.190395 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:42.191612 27965 hybrid_clock.cc:648] HybridClock initialized: now 1714702362191547 us; error 55 us; skew 500 ppm
I20240503 02:12:42.191944 27965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:42.194464 27965 webserver.cc:460] Webserver started at http://127.20.218.68:45885/ using document root <none> and password file <none>
I20240503 02:12:42.195274 27965 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:42.195384 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:42.195720 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:42.198568 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b0eb49c52e794992a8fc34173f0cdcef"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.199419 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b0eb49c52e794992a8fc34173f0cdcef"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.204993 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240503 02:12:42.209091 27979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.210876 27965 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:42.211076 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b0eb49c52e794992a8fc34173f0cdcef"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.211313 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:42.259323 27965 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:42.260248 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:42.260538 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:42.261582 27965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:42.263167 27965 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:42.263274 27965 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.263399 27965 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:42.263470 27965 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.296871 27965 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:39319
I20240503 02:12:42.297093 28091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:39319 every 8 connection(s)
I20240503 02:12:42.298370 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:42.303799 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 27965
I20240503 02:12:42.303999 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:12:42.307397 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:0
--local_ip_for_outbound_sockets=127.20.218.69
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:42.313635 28092 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:42.314044 28092 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:42.314877 28092 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:42.316558 27515 ts_manager.cc:194] Registered new tserver with Master: b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
I20240503 02:12:42.317848 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:46201
I20240503 02:12:42.333439 27832 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
W20240503 02:12:42.661834 28095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:42.662153 28095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:42.662279 28095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:42.670128 28095 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:42.670274 28095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:42.670446 28095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:12:42.679958 28095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:42.681448 28095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:42.683286 28095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:42.695940 28100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:42.696050 28101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:42.696050 28103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:42.697136 28095 server_base.cc:1034] running on GCE node
I20240503 02:12:42.697835 28095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:42.698865 28095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:42.700105 28095 hybrid_clock.cc:648] HybridClock initialized: now 1714702362700052 us; error 43 us; skew 500 ppm
I20240503 02:12:42.700466 28095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:42.703042 28095 webserver.cc:460] Webserver started at http://127.20.218.69:38999/ using document root <none> and password file <none>
I20240503 02:12:42.703841 28095 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:42.703951 28095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:42.704262 28095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:42.707118 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a8ef897d344843e2bbbbccc342f924ca"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.707994 28095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a8ef897d344843e2bbbbccc342f924ca"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.713881 28095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
I20240503 02:12:42.717886 28109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.719478 28095 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20240503 02:12:42.719664 28095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a8ef897d344843e2bbbbccc342f924ca"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:42.719907 28095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:42.735746 28095 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:42.737303 28095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:42.737710 28095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:42.738824 28095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:42.740520 28095 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:42.740633 28095 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.740753 28095 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:42.740823 28095 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:42.775143 28095 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:32881
I20240503 02:12:42.775192 28221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:32881 every 8 connection(s)
I20240503 02:12:42.776512 28095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:42.780045 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 28095
I20240503 02:12:42.780228 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:12:42.783449 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:0
--local_ip_for_outbound_sockets=127.20.218.70
--webserver_interface=127.20.218.70
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:42.791518 28222 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:42.791893 28222 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:42.792912 28222 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:42.794644 27515 ts_manager.cc:194] Registered new tserver with Master: a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881)
I20240503 02:12:42.795692 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:35569
I20240503 02:12:42.812047 27962 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
W20240503 02:12:43.131579 28225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:43.131922 28225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:43.132059 28225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:43.139990 28225 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:43.140157 28225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:43.140336 28225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:12:43.149729 28225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:43.151096 28225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:43.152870 28225 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:43.165040 28230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:43.165115 28231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:43.165887 28233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:43.166697 28225 server_base.cc:1034] running on GCE node
I20240503 02:12:43.167353 28225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:43.168442 28225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:43.169648 28225 hybrid_clock.cc:648] HybridClock initialized: now 1714702363169595 us; error 47 us; skew 500 ppm
I20240503 02:12:43.169993 28225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:43.172447 28225 webserver.cc:460] Webserver started at http://127.20.218.70:45303/ using document root <none> and password file <none>
I20240503 02:12:43.173245 28225 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:43.173418 28225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:43.173815 28225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:43.176614 28225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "24868c13491843548e8da6583ef1e9d5"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.177505 28225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "24868c13491843548e8da6583ef1e9d5"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.183125 28225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:43.186820 28239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.188479 28225 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:12:43.188689 28225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "24868c13491843548e8da6583ef1e9d5"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.188894 28225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:43.224684 28225 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:43.225960 28225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:43.226260 28225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:43.227296 28225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:43.228821 28225 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:43.228910 28225 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.229018 28225 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:43.229082 28225 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.267086 28225 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:37393
I20240503 02:12:43.267158 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:37393 every 8 connection(s)
I20240503 02:12:43.268553 28225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:43.276796 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 28225
I20240503 02:12:43.276958 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:12:43.280093 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:0
--local_ip_for_outbound_sockets=127.20.218.71
--webserver_interface=127.20.218.71
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:43.282454 28352 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:43.282766 28352 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:43.283572 28352 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:43.284991 27515 ts_manager.cc:194] Registered new tserver with Master: 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393)
I20240503 02:12:43.285840 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:56357
I20240503 02:12:43.320711 28092 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
W20240503 02:12:43.625924 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:43.626242 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:43.626344 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:43.634389 28355 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:43.634532 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:43.634667 28355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:12:43.644116 28355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:43.645553 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:43.647346 28355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:43.659719 28361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:43.659767 28360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:43.659684 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:43.660576 28355 server_base.cc:1034] running on GCE node
I20240503 02:12:43.661221 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:43.662261 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:43.663470 28355 hybrid_clock.cc:648] HybridClock initialized: now 1714702363663419 us; error 44 us; skew 500 ppm
I20240503 02:12:43.663798 28355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:43.666317 28355 webserver.cc:460] Webserver started at http://127.20.218.71:39761/ using document root <none> and password file <none>
I20240503 02:12:43.667111 28355 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:43.667232 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:43.667551 28355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:43.670274 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "94278d0a47dc47f2a0d5be9589f641be"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.671118 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "94278d0a47dc47f2a0d5be9589f641be"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.676617 28355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240503 02:12:43.680398 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.681942 28355 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:43.682140 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "94278d0a47dc47f2a0d5be9589f641be"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:43.682410 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:43.698403 28355 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:43.699331 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:43.699625 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:43.700680 28355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:43.702390 28355 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:43.702502 28355 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.702625 28355 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:43.702697 28355 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:43.735353 28355 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:40065
I20240503 02:12:43.735402 28481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:40065 every 8 connection(s)
I20240503 02:12:43.736900 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:43.741190 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 28355
I20240503 02:12:43.741431 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:12:43.750381 28482 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:43.750826 28482 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:43.751844 28482 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:43.753916 27515 ts_manager.cc:194] Registered new tserver with Master: 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065)
I20240503 02:12:43.754760 27515 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:54811
I20240503 02:12:43.758178 21353 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:12:43.776937 27515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:12:43.799307 28222 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:12:43.823401 27897 tablet_service.cc:1467] Processing CreateTablet for tablet 42ad89cd14fb4dcfbfa7caeb944549c2 (DEFAULT_TABLE table=TestTable [id=c3ebe602e0a946b19616e7756eaf7161]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.823376 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 42ad89cd14fb4dcfbfa7caeb944549c2 (DEFAULT_TABLE table=TestTable [id=c3ebe602e0a946b19616e7756eaf7161]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.825155 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ad89cd14fb4dcfbfa7caeb944549c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.825146 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ad89cd14fb4dcfbfa7caeb944549c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.840540 28498 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:43.840540 28499 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:43.844540 28499 tablet_bootstrap.cc:654] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.844540 28498 tablet_bootstrap.cc:654] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.846009 28498 log.cc:824] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:43.846009 28499 log.cc:824] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:43.849354 28499 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:43.849352 28498 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:43.849689 28499 ts_tablet_manager.cc:1367] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.009s	user 0.009s	sys 0.000s
I20240503 02:12:43.849686 28498 ts_tablet_manager.cc:1367] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.003s
I20240503 02:12:43.859459 28499 raft_consensus.cc:348] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:43.859459 28498 raft_consensus.cc:348] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:43.859994 28498 raft_consensus.cc:374] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.859997 28499 raft_consensus.cc:374] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.860128 28498 raft_consensus.cc:729] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.860127 28499 raft_consensus.cc:729] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.860996 28499 consensus_queue.cc:260] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:43.860996 28498 consensus_queue.cc:260] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:43.863162 28498 ts_tablet_manager.cc:1398] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.004s
I20240503 02:12:43.863515 28499 ts_tablet_manager.cc:1398] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.014s	user 0.014s	sys 0.000s
W20240503 02:12:44.044420 27963 tablet.cc:2369] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:44.064934 27833 tablet.cc:2369] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:44.154598 28503 raft_consensus.cc:482] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.154886 28503 raft_consensus.cc:504] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:44.156520 28503 leader_election.cc:288] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:44.163172 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:44.163704 27787 raft_consensus.cc:2454] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 0.
I20240503 02:12:44.164391 27850 leader_election.cc:302] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:44.164785 28503 raft_consensus.cc:2781] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.164949 28503 raft_consensus.cc:482] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.165053 28503 raft_consensus.cc:3037] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.167162 28503 raft_consensus.cc:504] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:44.167742 28503 leader_election.cc:288] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:44.168388 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:44.168576 27787 raft_consensus.cc:3037] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.170869 27787 raft_consensus.cc:2454] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 1.
I20240503 02:12:44.171389 27850 leader_election.cc:302] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:44.171762 28503 raft_consensus.cc:2781] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.172382 28503 raft_consensus.cc:686] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Becoming Leader. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:12:44.172902 28503 consensus_queue.cc:237] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:44.177469 27515 catalog_manager.cc:5484] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } }
I20240503 02:12:44.268587 21353 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:12:44.271023 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ecf926bd3fb84e0d8f66465d78a70d31 to finish bootstrapping
I20240503 02:12:44.288447 28352 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:12:44.288691 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver acd475a9eee143a89c1d28a05fa51d26 to finish bootstrapping
I20240503 02:12:44.297118 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eaffa047cf9b43b79a5b75089ea63901 to finish bootstrapping
I20240503 02:12:44.305739 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b0eb49c52e794992a8fc34173f0cdcef to finish bootstrapping
I20240503 02:12:44.315107 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a8ef897d344843e2bbbbccc342f924ca to finish bootstrapping
I20240503 02:12:44.324941 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 24868c13491843548e8da6583ef1e9d5 to finish bootstrapping
I20240503 02:12:44.334432 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 94278d0a47dc47f2a0d5be9589f641be to finish bootstrapping
I20240503 02:12:44.343611 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27965
I20240503 02:12:44.344695 28087 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:44.429157 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27965
I20240503 02:12:44.445732 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 28095
I20240503 02:12:44.447072 28217 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:44.529222 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 28095
I20240503 02:12:44.545758 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 28225
I20240503 02:12:44.546841 28347 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:44.628276 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 28225
I20240503 02:12:44.645366 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 28355
I20240503 02:12:44.646617 28477 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:44.719309 28517 consensus_queue.cc:1035] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.730268 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 28355
I20240503 02:12:48.498044 27515 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:48.507920 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 878e847516c74149966db83f6f70c6d0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.508461 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878e847516c74149966db83f6f70c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.509500 27766 tablet_service.cc:1467] Processing CreateTablet for tablet a52a6040dd89493da32af4b283774f03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.509671 27897 tablet_service.cc:1467] Processing CreateTablet for tablet 9698b9fc77ea4172b58f410383f2d926 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.509938 27766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52a6040dd89493da32af4b283774f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.510071 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9698b9fc77ea4172b58f410383f2d926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.517782 28532 tablet_bootstrap.cc:492] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:48.518354 28531 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:48.519484 28532 tablet_bootstrap.cc:654] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.520335 28531 tablet_bootstrap.cc:654] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.521014 27636 tablet_service.cc:1467] Processing CreateTablet for tablet 878e847516c74149966db83f6f70c6d0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.521052 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 9698b9fc77ea4172b58f410383f2d926 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.521948 27634 tablet_service.cc:1467] Processing CreateTablet for tablet a52a6040dd89493da32af4b283774f03 (DEFAULT_TABLE table=rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:48.522862 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9698b9fc77ea4172b58f410383f2d926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.523293 28532 tablet_bootstrap.cc:492] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:48.523322 28531 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:48.523468 28532 ts_tablet_manager.cc:1367] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240503 02:12:48.523470 28531 ts_tablet_manager.cc:1367] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:12:48.523599 27636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878e847516c74149966db83f6f70c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.524439 28532 raft_consensus.cc:348] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.524680 28532 raft_consensus.cc:374] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.524791 28532 raft_consensus.cc:729] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.524696 28531 raft_consensus.cc:348] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.524839 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52a6040dd89493da32af4b283774f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:48.524914 28531 raft_consensus.cc:374] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.525012 28531 raft_consensus.cc:729] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.525038 28532 consensus_queue.cc:260] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.525255 28531 consensus_queue.cc:260] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.525615 28532 ts_tablet_manager.cc:1398] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:48.526160 28531 ts_tablet_manager.cc:1398] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:12:48.526438 28531 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:48.528003 28531 tablet_bootstrap.cc:654] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.531275 28531 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:48.531451 28531 ts_tablet_manager.cc:1367] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:48.532485 28531 raft_consensus.cc:348] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.532709 28531 raft_consensus.cc:374] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.532788 28531 raft_consensus.cc:729] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.533098 28531 consensus_queue.cc:260] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.533741 28531 ts_tablet_manager.cc:1398] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:12:48.539553 28537 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:48.542696 28537 tablet_bootstrap.cc:654] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.543751 28537 log.cc:824] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:48.546561 28537 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:48.546836 28537 ts_tablet_manager.cc:1367] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240503 02:12:48.553292 28537 raft_consensus.cc:348] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.553714 28537 raft_consensus.cc:374] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.553807 28537 raft_consensus.cc:729] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.554404 28537 consensus_queue.cc:260] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.555768 28537 ts_tablet_manager.cc:1398] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
I20240503 02:12:48.556118 28537 tablet_bootstrap.cc:492] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:48.557685 28537 tablet_bootstrap.cc:654] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.559629 28537 tablet_bootstrap.cc:492] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:48.559751 28537 ts_tablet_manager.cc:1367] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:48.560518 28537 raft_consensus.cc:348] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.560714 28537 raft_consensus.cc:374] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.560817 28537 raft_consensus.cc:729] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.561053 28537 consensus_queue.cc:260] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.561518 28537 ts_tablet_manager.cc:1398] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:48.561777 28537 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:48.563431 28537 tablet_bootstrap.cc:654] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:48.565418 28537 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:48.565564 28537 ts_tablet_manager.cc:1367] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:48.566118 28537 raft_consensus.cc:348] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.566313 28537 raft_consensus.cc:374] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:48.566385 28537 raft_consensus.cc:729] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:48.566622 28537 consensus_queue.cc:260] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.567090 28537 ts_tablet_manager.cc:1398] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
W20240503 02:12:48.577692 27702 tablet.cc:2369] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:48.683202 28539 raft_consensus.cc:482] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.683461 28539 raft_consensus.cc:504] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.685026 28539 leader_election.cc:288] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:48.691004 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878e847516c74149966db83f6f70c6d0" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:48.691262 27787 raft_consensus.cc:2454] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:48.691816 27589 leader_election.cc:302] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.692209 28539 raft_consensus.cc:2781] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.692378 28539 raft_consensus.cc:482] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.692478 28539 raft_consensus.cc:3037] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.694698 28539 raft_consensus.cc:504] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.695230 28539 leader_election.cc:288] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:48.695843 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878e847516c74149966db83f6f70c6d0" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:48.696074 27787 raft_consensus.cc:3037] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.698053 27787 raft_consensus.cc:2454] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:48.698513 27589 leader_election.cc:302] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.698835 28539 raft_consensus.cc:2781] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.699298 28539 raft_consensus.cc:686] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:48.699819 28539 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.704015 27513 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:48.729902 28539 raft_consensus.cc:482] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.730147 28539 raft_consensus.cc:504] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.730758 28539 leader_election.cc:288] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:48.736691 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9698b9fc77ea4172b58f410383f2d926" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:48.737071 27917 raft_consensus.cc:2454] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:48.737593 27592 leader_election.cc:302] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.737908 28539 raft_consensus.cc:2781] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.737999 28539 raft_consensus.cc:482] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.738107 28539 raft_consensus.cc:3037] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.740015 28539 raft_consensus.cc:504] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.740525 28539 leader_election.cc:288] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:48.741091 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9698b9fc77ea4172b58f410383f2d926" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:48.741348 27917 raft_consensus.cc:3037] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.743324 27917 raft_consensus.cc:2454] T 9698b9fc77ea4172b58f410383f2d926 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:48.743716 27592 leader_election.cc:302] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.743970 28539 raft_consensus.cc:2781] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.744138 28539 raft_consensus.cc:686] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:48.744398 28539 consensus_queue.cc:237] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:48.746783 27513 catalog_manager.cc:5484] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:48.843230 28535 raft_consensus.cc:482] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:48.843452 28535 raft_consensus.cc:504] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.844883 28535 leader_election.cc:288] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:48.851152 27656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52a6040dd89493da32af4b283774f03" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" is_pre_election: true
I20240503 02:12:48.851576 27656 raft_consensus.cc:2454] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 0.
I20240503 02:12:48.852209 27722 leader_election.cc:302] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.852604 28535 raft_consensus.cc:2781] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:48.852746 28535 raft_consensus.cc:482] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:48.852845 28535 raft_consensus.cc:3037] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.854915 28535 raft_consensus.cc:504] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.855405 28535 leader_election.cc:288] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:48.855978 27656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52a6040dd89493da32af4b283774f03" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:12:48.856186 27656 raft_consensus.cc:3037] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:48.858304 27656 raft_consensus.cc:2454] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 1.
I20240503 02:12:48.858774 27722 leader_election.cc:302] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:48.859094 28535 raft_consensus.cc:2781] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:48.859510 28535 raft_consensus.cc:686] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:12:48.860002 28535 consensus_queue.cc:237] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:48.862905 27513 catalog_manager.cc:5484] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } }
I20240503 02:12:49.146420 28539 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:49.232825 28551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:49.233189 28551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:49.241547 28551 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 02:12:49.295857 28535 consensus_queue.cc:1035] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:49.335850 28544 consensus_queue.cc:1035] T 9698b9fc77ea4172b58f410383f2d926 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 192.86 ms
  time per row: 0.0964302 ms
I20240503 02:12:49.583114 27513 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:49.592516 27897 tablet_service.cc:1467] Processing CreateTablet for tablet 6b842b202e3b4d17afd245f0d90a34a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.593021 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b842b202e3b4d17afd245f0d90a34a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.593190 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 6b842b202e3b4d17afd245f0d90a34a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.593696 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b842b202e3b4d17afd245f0d90a34a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.593564 27895 tablet_service.cc:1467] Processing CreateTablet for tablet 4395a743162549ca96757c5838c0e190 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.594091 27895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395a743162549ca96757c5838c0e190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.598572 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec65a7b8d0943119b929ae3a9544ec0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.599078 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec65a7b8d0943119b929ae3a9544ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.600426 28601 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:49.600471 27896 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec65a7b8d0943119b929ae3a9544ec0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.601106 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec65a7b8d0943119b929ae3a9544ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.602792 28601 tablet_bootstrap.cc:654] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.604337 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 4395a743162549ca96757c5838c0e190 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:49.604808 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395a743162549ca96757c5838c0e190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.610916 28603 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:49.613617 28603 tablet_bootstrap.cc:654] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.619104 28604 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:49.620033 28601 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:49.620184 28601 ts_tablet_manager.cc:1367] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.011s
I20240503 02:12:49.620798 28604 tablet_bootstrap.cc:654] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.620947 28601 raft_consensus.cc:348] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.621161 28601 raft_consensus.cc:374] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.621251 28601 raft_consensus.cc:729] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.621286 28603 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:49.621510 28603 ts_tablet_manager.cc:1367] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.009s
I20240503 02:12:49.621529 28601 consensus_queue.cc:260] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.622152 28601 ts_tablet_manager.cc:1398] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:49.622342 28603 raft_consensus.cc:348] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.622541 28601 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:49.622534 28603 raft_consensus.cc:374] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.622624 28603 raft_consensus.cc:729] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.623142 28603 consensus_queue.cc:260] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.623747 28603 ts_tablet_manager.cc:1398] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:12:49.625228 28601 tablet_bootstrap.cc:654] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.625659 28604 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:49.625821 28604 ts_tablet_manager.cc:1367] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240503 02:12:49.626626 28604 raft_consensus.cc:348] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.626847 28604 raft_consensus.cc:374] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.626941 28604 raft_consensus.cc:729] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.627174 28604 consensus_queue.cc:260] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.627635 28604 ts_tablet_manager.cc:1398] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:49.628178 28604 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:49.629931 28604 tablet_bootstrap.cc:654] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.630754 28601 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:49.630998 28601 ts_tablet_manager.cc:1367] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:12:49.631628 28601 raft_consensus.cc:348] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.631811 28601 raft_consensus.cc:374] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.631896 28601 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.632134 28601 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.632684 28601 ts_tablet_manager.cc:1398] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:49.633034 28601 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:49.633551 28604 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:49.633811 28604 ts_tablet_manager.cc:1367] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:12:49.634447 28601 tablet_bootstrap.cc:654] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.634665 28604 raft_consensus.cc:348] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.634824 28604 raft_consensus.cc:374] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.634889 28604 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.635133 28604 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.635707 28604 ts_tablet_manager.cc:1398] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:49.642706 28601 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:49.642871 28601 ts_tablet_manager.cc:1367] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.005s
I20240503 02:12:49.643528 28601 raft_consensus.cc:348] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.643716 28601 raft_consensus.cc:374] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:49.643810 28601 raft_consensus.cc:729] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:49.644042 28601 consensus_queue.cc:260] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.644560 28601 ts_tablet_manager.cc:1398] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:49.672755 28544 raft_consensus.cc:482] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.672966 28544 raft_consensus.cc:504] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.673553 28544 leader_election.cc:288] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.674139 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:49.674365 27917 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:49.674768 27592 leader_election.cc:302] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:49.675067 28544 raft_consensus.cc:2781] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.675164 28544 raft_consensus.cc:482] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.675235 28544 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.677136 28544 raft_consensus.cc:504] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.677721 28544 leader_election.cc:288] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.678542 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:49.678829 27917 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.680824 27917 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:49.681398 27592 leader_election.cc:302] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:49.681768 28544 raft_consensus.cc:2781] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.681951 28544 raft_consensus.cc:686] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:49.682217 28544 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.685361 27515 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:49.747051 28535 raft_consensus.cc:482] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.747254 28535 raft_consensus.cc:504] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.747905 28535 leader_election.cc:288] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.755558 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:49.755851 27917 raft_consensus.cc:2454] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 0.
I20240503 02:12:49.756414 27723 leader_election.cc:302] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:49.756767 28535 raft_consensus.cc:2781] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.756906 28535 raft_consensus.cc:482] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.756999 28535 raft_consensus.cc:3037] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.758978 28535 raft_consensus.cc:504] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.759528 28535 leader_election.cc:288] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.760402 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:49.760715 27917 raft_consensus.cc:3037] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.763561 27917 raft_consensus.cc:2454] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 1.
I20240503 02:12:49.764118 27723 leader_election.cc:302] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:49.764462 28535 raft_consensus.cc:2781] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.764618 28535 raft_consensus.cc:686] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:12:49.764936 28535 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:49.767827 27515 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } }
I20240503 02:12:49.931106 28544 raft_consensus.cc:482] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:49.931325 28544 raft_consensus.cc:504] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.931984 28544 leader_election.cc:288] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.932641 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b842b202e3b4d17afd245f0d90a34a4" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:49.932931 27917 raft_consensus.cc:2454] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:49.933424 27592 leader_election.cc:302] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:49.933722 28544 raft_consensus.cc:2781] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:49.933825 28544 raft_consensus.cc:482] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:49.933935 28544 raft_consensus.cc:3037] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.935878 28544 raft_consensus.cc:504] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.936444 28544 leader_election.cc:288] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:49.936951 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b842b202e3b4d17afd245f0d90a34a4" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:49.937273 27916 raft_consensus.cc:3037] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:49.939469 27916 raft_consensus.cc:2454] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:49.939991 27592 leader_election.cc:302] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:49.940359 28544 raft_consensus.cc:2781] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:49.940541 28544 raft_consensus.cc:686] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:49.940871 28544 consensus_queue.cc:237] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:49.943723 27515 catalog_manager.cc:5484] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:50.230010 28544 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:50.255898 28535 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:50.329052 28615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:50.329427 28615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:50.337260 28615 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:12:50.506317 27916 raft_consensus.cc:1261] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:50.508991 28539 consensus_queue.cc:1035] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:50.527488 28617 mvcc.cc:205] Tried to move back new op lower bound from 7023420909598523392 to 7023420907278127104. Current Snapshot: MvccSnapshot[applied={T|T < 7023420909570629632}]

INSERT report
    rows total: 2000
    time total: 180.58 ms
  time per row: 0.0902901 ms
I20240503 02:12:50.581708 28618 mvcc.cc:205] Tried to move back new op lower bound from 7023420909598523392 to 7023420907278127104. Current Snapshot: MvccSnapshot[applied={T|T < 7023420909570629632}]
I20240503 02:12:50.675823 27507 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:50.685698 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 376511df6d8645268134e125791f6aca (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.685866 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 376511df6d8645268134e125791f6aca (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.686118 27766 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd7e1c1b87f4f6db714735a13e25f67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.686265 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376511df6d8645268134e125791f6aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.686276 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376511df6d8645268134e125791f6aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.686496 27766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd7e1c1b87f4f6db714735a13e25f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.686841 27897 tablet_service.cc:1467] Processing CreateTablet for tablet 4dd7e1c1b87f4f6db714735a13e25f67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.687265 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd7e1c1b87f4f6db714735a13e25f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.694782 27634 tablet_service.cc:1467] Processing CreateTablet for tablet a63be7e6f2da4163baced11894065444 (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.695494 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63be7e6f2da4163baced11894065444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.695470 27765 tablet_service.cc:1467] Processing CreateTablet for tablet a63be7e6f2da4163baced11894065444 (DEFAULT_TABLE table=rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.696278 27765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63be7e6f2da4163baced11894065444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.699465 28667 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:50.702795 28668 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:50.704857 28668 tablet_bootstrap.cc:654] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.705811 28669 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:50.706684 28667 tablet_bootstrap.cc:654] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.708120 28669 tablet_bootstrap.cc:654] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.710848 28668 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:50.710951 28669 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:50.711017 28668 ts_tablet_manager.cc:1367] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240503 02:12:50.711097 28667 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:50.711166 28669 ts_tablet_manager.cc:1367] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:12:50.711248 28667 ts_tablet_manager.cc:1367] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.002s
I20240503 02:12:50.711745 28668 raft_consensus.cc:348] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.711921 28668 raft_consensus.cc:374] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.712011 28668 raft_consensus.cc:729] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.712010 28667 raft_consensus.cc:348] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.711999 28669 raft_consensus.cc:348] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:50.712205 28667 raft_consensus.cc:374] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.712209 28669 raft_consensus.cc:374] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.712297 28667 raft_consensus.cc:729] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.712307 28669 raft_consensus.cc:729] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.712299 28668 consensus_queue.cc:260] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.712530 28669 consensus_queue.cc:260] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:50.712522 28667 consensus_queue.cc:260] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.712846 28668 ts_tablet_manager.cc:1398] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:12:50.713105 28667 ts_tablet_manager.cc:1398] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:50.713152 28668 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:50.714205 28667 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:50.714890 28668 tablet_bootstrap.cc:654] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.715948 28667 tablet_bootstrap.cc:654] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.724618 28669 ts_tablet_manager.cc:1398] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240503 02:12:50.727073 28668 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:50.727232 28667 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:50.727293 28668 ts_tablet_manager.cc:1367] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.002s
I20240503 02:12:50.727423 28667 ts_tablet_manager.cc:1367] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.004s
I20240503 02:12:50.728173 28668 raft_consensus.cc:348] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:50.728369 28668 raft_consensus.cc:374] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.728447 28668 raft_consensus.cc:729] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.728628 28667 raft_consensus.cc:348] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.728684 28668 consensus_queue.cc:260] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:50.728842 28667 raft_consensus.cc:374] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.728935 28667 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.729213 28668 ts_tablet_manager.cc:1398] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:12:50.729200 28667 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.729609 28668 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:50.729697 28667 ts_tablet_manager.cc:1398] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:12:50.731732 28668 tablet_bootstrap.cc:654] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.734088 28668 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:50.734234 28668 ts_tablet_manager.cc:1367] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:12:50.734845 28668 raft_consensus.cc:348] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.735025 28668 raft_consensus.cc:374] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.735118 28668 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.735325 28668 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.735800 28668 ts_tablet_manager.cc:1398] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:12:50.735868 28656 raft_consensus.cc:482] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.736037 28656 raft_consensus.cc:504] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.736634 28656 leader_election.cc:288] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:50.737118 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376511df6d8645268134e125791f6aca" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:50.737357 27787 raft_consensus.cc:2454] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:50.738399 27589 leader_election.cc:302] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:50.738766 28656 raft_consensus.cc:2781] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:50.738866 28656 raft_consensus.cc:482] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:50.738974 28656 raft_consensus.cc:3037] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.741241 28656 raft_consensus.cc:504] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.741765 28656 leader_election.cc:288] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:50.742372 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376511df6d8645268134e125791f6aca" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:50.742569 27787 raft_consensus.cc:3037] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.744488 27787 raft_consensus.cc:2454] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:50.744951 27589 leader_election.cc:302] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:50.745321 28656 raft_consensus.cc:2781] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:50.745595 28656 raft_consensus.cc:686] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:50.745879 28656 consensus_queue.cc:237] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.749042 27507 catalog_manager.cc:5484] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:50.998241 28656 raft_consensus.cc:482] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.998474 28656 raft_consensus.cc:504] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:50.998982 28656 leader_election.cc:288] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:50.999557 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:50.999799 27787 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:51.000285 27589 leader_election.cc:302] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:51.000660 28656 raft_consensus.cc:2781] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.000758 28656 raft_consensus.cc:482] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.000890 28656 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.002998 28656 raft_consensus.cc:504] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:51.003533 28656 leader_election.cc:288] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:51.004153 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:51.004374 27787 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.006374 27787 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:51.007107 27589 leader_election.cc:302] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:51.007470 28656 raft_consensus.cc:2781] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.007731 28656 raft_consensus.cc:686] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:51.008047 28656 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:51.011745 27507 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:51.114218 28549 raft_consensus.cc:482] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.114416 28549 raft_consensus.cc:504] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.115013 28549 leader_election.cc:288] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.115602 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:51.115828 27916 raft_consensus.cc:2454] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 0.
I20240503 02:12:51.116513 27723 leader_election.cc:302] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:51.116847 28549 raft_consensus.cc:2781] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.116968 28549 raft_consensus.cc:482] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.117038 28549 raft_consensus.cc:3037] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.118966 28549 raft_consensus.cc:504] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.119462 28549 leader_election.cc:288] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.120142 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:51.120360 27916 raft_consensus.cc:3037] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.122547 27916 raft_consensus.cc:2454] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 1.
I20240503 02:12:51.123023 27723 leader_election.cc:302] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:51.123422 28549 raft_consensus.cc:2781] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.123785 28549 raft_consensus.cc:686] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:12:51.124063 28549 consensus_queue.cc:237] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.126673 27507 catalog_manager.cc:5484] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } }
I20240503 02:12:51.306529 28656 consensus_queue.cc:1035] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.512032 28656 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:51.520520 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:51.520917 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:51.529009 28680 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:12:51.608938 28549 consensus_queue.cc:1035] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 159.414 ms
  time per row: 0.0797069 ms
I20240503 02:12:51.843303 27509 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:51.852447 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc58692b4d144ed8bd4ad33cc8d022c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.852596 27897 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc58692b4d144ed8bd4ad33cc8d022c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.852840 27896 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf2a0fe803b42e4a706f50762a4b87d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.853004 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc58692b4d144ed8bd4ad33cc8d022c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.853051 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc58692b4d144ed8bd4ad33cc8d022c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.853437 27896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf2a0fe803b42e4a706f50762a4b87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.853596 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf2a0fe803b42e4a706f50762a4b87d (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.854010 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf2a0fe803b42e4a706f50762a4b87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.858549 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 896ae927d34e46188120e705fb0ae361 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.859014 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896ae927d34e46188120e705fb0ae361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.860037 28730 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:51.861070 27895 tablet_service.cc:1467] Processing CreateTablet for tablet 896ae927d34e46188120e705fb0ae361 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.861546 27895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896ae927d34e46188120e705fb0ae361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.863401 28731 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:51.865188 28730 tablet_bootstrap.cc:654] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.865360 28731 tablet_bootstrap.cc:654] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.868090 28730 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:51.868484 28730 ts_tablet_manager.cc:1367] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:12:51.869717 28730 raft_consensus.cc:348] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.870011 28730 raft_consensus.cc:374] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.870141 28730 raft_consensus.cc:729] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.870497 28730 consensus_queue.cc:260] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.870985 28734 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:51.872236 28730 ts_tablet_manager.cc:1398] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:12:51.872356 28731 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:51.872514 28731 ts_tablet_manager.cc:1367] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240503 02:12:51.872536 28730 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:51.873088 28734 tablet_bootstrap.cc:654] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.873375 28731 raft_consensus.cc:348] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.873610 28731 raft_consensus.cc:374] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.873706 28731 raft_consensus.cc:729] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.873955 28731 consensus_queue.cc:260] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.874586 28731 ts_tablet_manager.cc:1398] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:12:51.874883 28731 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:51.875708 28730 tablet_bootstrap.cc:654] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.876590 28731 tablet_bootstrap.cc:654] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.878554 28734 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:51.878571 28730 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:51.878763 28730 ts_tablet_manager.cc:1367] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.004s
I20240503 02:12:51.878899 28734 ts_tablet_manager.cc:1367] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240503 02:12:51.879595 28730 raft_consensus.cc:348] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.879842 28730 raft_consensus.cc:374] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.879933 28730 raft_consensus.cc:729] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.880174 28730 consensus_queue.cc:260] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.880926 28730 ts_tablet_manager.cc:1398] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:51.881541 28734 raft_consensus.cc:348] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.881929 28734 raft_consensus.cc:374] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.882046 28734 raft_consensus.cc:729] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.882292 28734 consensus_queue.cc:260] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.882609 28731 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:51.882789 28731 ts_tablet_manager.cc:1367] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240503 02:12:51.882979 28734 ts_tablet_manager.cc:1398] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:12:51.883560 28731 raft_consensus.cc:348] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.883785 28731 raft_consensus.cc:374] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.883849 28731 raft_consensus.cc:729] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.884052 28731 consensus_queue.cc:260] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.884547 28731 ts_tablet_manager.cc:1398] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:51.884867 28731 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:51.886485 28731 tablet_bootstrap.cc:654] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.888375 28731 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:51.888530 28731 ts_tablet_manager.cc:1367] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:51.889264 28731 raft_consensus.cc:348] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.889493 28731 raft_consensus.cc:374] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.889590 28731 raft_consensus.cc:729] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.889830 28731 consensus_queue.cc:260] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.890275 28731 ts_tablet_manager.cc:1398] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:51.910576 28656 raft_consensus.cc:482] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.910832 28656 raft_consensus.cc:504] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.911396 28656 leader_election.cc:288] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.911990 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896ae927d34e46188120e705fb0ae361" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:51.912268 27916 raft_consensus.cc:2454] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:51.912700 27592 leader_election.cc:302] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:51.912971 28656 raft_consensus.cc:2781] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.913067 28656 raft_consensus.cc:482] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.913161 28656 raft_consensus.cc:3037] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.913544 28549 raft_consensus.cc:482] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.913686 28549 raft_consensus.cc:504] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.914201 28549 leader_election.cc:288] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.914703 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf2a0fe803b42e4a706f50762a4b87d" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901" is_pre_election: true
I20240503 02:12:51.914918 27916 raft_consensus.cc:2454] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 0.
I20240503 02:12:51.915310 27723 leader_election.cc:302] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:51.915275 28656 raft_consensus.cc:504] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.915560 28549 raft_consensus.cc:2781] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.915689 28549 raft_consensus.cc:482] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.915791 28549 raft_consensus.cc:3037] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.916002 28656 leader_election.cc:288] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.916270 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896ae927d34e46188120e705fb0ae361" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:51.916463 27916 raft_consensus.cc:3037] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.917837 28549 raft_consensus.cc:504] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.918313 28549 leader_election.cc:288] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:51.918453 27916 raft_consensus.cc:2454] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:51.918895 27592 leader_election.cc:302] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:51.918888 27916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf2a0fe803b42e4a706f50762a4b87d" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:12:51.919056 27916 raft_consensus.cc:3037] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.919230 28656 raft_consensus.cc:2781] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.919385 28656 raft_consensus.cc:686] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:51.919660 28656 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:51.921509 27916 raft_consensus.cc:2454] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 1.
I20240503 02:12:51.922096 27723 leader_election.cc:302] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:51.922453 28549 raft_consensus.cc:2781] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.922618 28549 raft_consensus.cc:686] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:12:51.922917 28549 consensus_queue.cc:237] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:51.922950 27514 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } }
I20240503 02:12:51.926307 27509 catalog_manager.cc:5484] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } }
I20240503 02:12:52.084941 28524 raft_consensus.cc:482] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:52.085175 28524 raft_consensus.cc:504] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:52.085942 28524 leader_election.cc:288] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:52.091905 27656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc58692b4d144ed8bd4ad33cc8d022c" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" is_pre_election: true
I20240503 02:12:52.092176 27656 raft_consensus.cc:2454] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 0.
I20240503 02:12:52.092702 27852 leader_election.cc:302] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:52.093000 28524 raft_consensus.cc:2781] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:52.093137 28524 raft_consensus.cc:482] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:52.093236 28524 raft_consensus.cc:3037] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.095758 28524 raft_consensus.cc:504] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:52.096282 28524 leader_election.cc:288] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:52.096788 27656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc58692b4d144ed8bd4ad33cc8d022c" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:12:52.097021 27656 raft_consensus.cc:3037] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.099030 27656 raft_consensus.cc:2454] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 1.
I20240503 02:12:52.099542 27852 leader_election.cc:302] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: eaffa047cf9b43b79a5b75089ea63901, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:52.099862 28524 raft_consensus.cc:2781] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:52.100263 28524 raft_consensus.cc:686] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Becoming Leader. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:12:52.100648 28524 consensus_queue.cc:237] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } }
I20240503 02:12:52.103493 27509 catalog_manager.cc:5484] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } }
W20240503 02:12:52.481494 28744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:52.481976 28744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:52.482302 28549 consensus_queue.cc:1035] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:52.492462 28744 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:12:52.533344 28544 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.674964 28524 consensus_queue.cc:1035] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 255.713 ms
  time per row: 0.127857 ms
I20240503 02:12:52.918902 27508 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50632:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:12:52.928465 27895 tablet_service.cc:1467] Processing CreateTablet for tablet f475da5fc141450ba4ddfe739ea3ad96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.928439 27767 tablet_service.cc:1467] Processing CreateTablet for tablet 6af88acfe24e49279a98b5dcf417bed7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.928568 27635 tablet_service.cc:1467] Processing CreateTablet for tablet 6af88acfe24e49279a98b5dcf417bed7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.928898 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af88acfe24e49279a98b5dcf417bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.928898 27895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f475da5fc141450ba4ddfe739ea3ad96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.929029 27635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af88acfe24e49279a98b5dcf417bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.929500 27634 tablet_service.cc:1467] Processing CreateTablet for tablet ddd409a1c4554665831eceef070ab660 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.929883 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd409a1c4554665831eceef070ab660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.936594 28803 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: Bootstrap starting.
I20240503 02:12:52.936993 27766 tablet_service.cc:1467] Processing CreateTablet for tablet ddd409a1c4554665831eceef070ab660 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.936955 27765 tablet_service.cc:1467] Processing CreateTablet for tablet f475da5fc141450ba4ddfe739ea3ad96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.937484 27766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd409a1c4554665831eceef070ab660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.938052 27765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f475da5fc141450ba4ddfe739ea3ad96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.939114 28803 tablet_bootstrap.cc:654] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.945444 28805 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:52.946012 28804 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:52.947436 28805 tablet_bootstrap.cc:654] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.948499 28803 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: No bootstrap required, opened a new log
I20240503 02:12:52.948550 28804 tablet_bootstrap.cc:654] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.948684 28803 ts_tablet_manager.cc:1367] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240503 02:12:52.949661 28803 raft_consensus.cc:348] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:52.949836 28805 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:52.949880 28803 raft_consensus.cc:374] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.949975 28803 raft_consensus.cc:729] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.949988 28805 ts_tablet_manager.cc:1367] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:12:52.950269 28803 consensus_queue.cc:260] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:52.950618 28805 raft_consensus.cc:348] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.950857 28804 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:52.950884 28805 raft_consensus.cc:374] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.951022 28804 ts_tablet_manager.cc:1367] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240503 02:12:52.951067 28803 ts_tablet_manager.cc:1398] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:12:52.951071 28805 raft_consensus.cc:729] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.951462 28805 consensus_queue.cc:260] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.952085 28805 ts_tablet_manager.cc:1398] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:52.952510 28805 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:52.953747 28804 raft_consensus.cc:348] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.953975 28804 raft_consensus.cc:374] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.954078 28804 raft_consensus.cc:729] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.954392 28804 consensus_queue.cc:260] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.954978 28805 tablet_bootstrap.cc:654] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.958101 28804 ts_tablet_manager.cc:1398] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240503 02:12:52.958352 28805 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:52.958487 28804 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Bootstrap starting.
I20240503 02:12:52.958529 28805 ts_tablet_manager.cc:1367] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240503 02:12:52.959287 28805 raft_consensus.cc:348] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:52.959543 28805 raft_consensus.cc:374] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.959621 28805 raft_consensus.cc:729] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.959897 28805 consensus_queue.cc:260] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:52.960289 28804 tablet_bootstrap.cc:654] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.960489 28805 ts_tablet_manager.cc:1398] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:52.960887 28805 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: Bootstrap starting.
I20240503 02:12:52.962436 28805 tablet_bootstrap.cc:654] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.962623 28804 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: No bootstrap required, opened a new log
I20240503 02:12:52.962795 28804 ts_tablet_manager.cc:1367] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:52.963754 28804 raft_consensus.cc:348] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.964046 28804 raft_consensus.cc:374] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.964143 28804 raft_consensus.cc:729] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.964496 28804 consensus_queue.cc:260] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.965257 28804 ts_tablet_manager.cc:1398] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:12:52.966555 28805 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: No bootstrap required, opened a new log
I20240503 02:12:52.966701 28805 ts_tablet_manager.cc:1367] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:12:52.966933 28656 raft_consensus.cc:482] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:52.967111 28656 raft_consensus.cc:504] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.967489 28805 raft_consensus.cc:348] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.967669 28805 raft_consensus.cc:374] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.967687 28656 leader_election.cc:288] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:52.968009 28805 raft_consensus.cc:729] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.968233 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af88acfe24e49279a98b5dcf417bed7" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:52.968331 28805 consensus_queue.cc:260] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.968449 27786 raft_consensus.cc:2454] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 0.
I20240503 02:12:52.968854 27589 leader_election.cc:302] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:52.969028 28805 ts_tablet_manager.cc:1398] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:12:52.969349 28656 raft_consensus.cc:2781] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:52.969467 28656 raft_consensus.cc:482] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:52.969594 28656 raft_consensus.cc:3037] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.971634 28656 raft_consensus.cc:504] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.972100 28656 leader_election.cc:288] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:52.972637 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af88acfe24e49279a98b5dcf417bed7" candidate_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:52.972877 27786 raft_consensus.cc:3037] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.975327 27786 raft_consensus.cc:2454] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecf926bd3fb84e0d8f66465d78a70d31 in term 1.
I20240503 02:12:52.975776 27589 leader_election.cc:302] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:52.976233 28656 raft_consensus.cc:2781] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:52.976747 28656 raft_consensus.cc:686] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Leader. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:12:52.977041 28656 consensus_queue.cc:237] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:52.979638 27507 catalog_manager.cc:5484] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:53.054886 28524 raft_consensus.cc:482] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.055090 28524 raft_consensus.cc:504] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:53.055719 28524 leader_election.cc:288] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:53.056335 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f475da5fc141450ba4ddfe739ea3ad96" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26" is_pre_election: true
I20240503 02:12:53.056569 27786 raft_consensus.cc:2454] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 0.
I20240503 02:12:53.057035 27850 leader_election.cc:302] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:53.057538 28524 raft_consensus.cc:2781] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.057683 28524 raft_consensus.cc:482] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.057806 28524 raft_consensus.cc:3037] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.060256 28524 raft_consensus.cc:504] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:53.060735 28524 leader_election.cc:288] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:12:53.061283 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f475da5fc141450ba4ddfe739ea3ad96" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:12:53.061550 27786 raft_consensus.cc:3037] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.063614 27786 raft_consensus.cc:2454] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 1.
I20240503 02:12:53.064051 27850 leader_election.cc:302] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:12:53.064391 28524 raft_consensus.cc:2781] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.064566 28524 raft_consensus.cc:686] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Becoming Leader. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:12:53.064860 28524 consensus_queue.cc:237] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } }
I20240503 02:12:53.067638 27507 catalog_manager.cc:5484] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } }
I20240503 02:12:53.161108 28549 raft_consensus.cc:482] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.161309 28549 raft_consensus.cc:504] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:53.161912 28549 leader_election.cc:288] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:53.162554 27655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd409a1c4554665831eceef070ab660" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" is_pre_election: true
I20240503 02:12:53.162776 27655 raft_consensus.cc:2454] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 0.
I20240503 02:12:53.163228 27722 leader_election.cc:302] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:53.163501 28549 raft_consensus.cc:2781] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.163606 28549 raft_consensus.cc:482] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.163697 28549 raft_consensus.cc:3037] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.165752 28549 raft_consensus.cc:504] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:53.166234 28549 leader_election.cc:288] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:12:53.166760 27655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd409a1c4554665831eceef070ab660" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:12:53.166963 27655 raft_consensus.cc:3037] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.168982 27655 raft_consensus.cc:2454] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 1.
I20240503 02:12:53.169448 27722 leader_election.cc:302] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:12:53.169772 28549 raft_consensus.cc:2781] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.169924 28549 raft_consensus.cc:686] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:12:53.170220 28549 consensus_queue.cc:237] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } }
I20240503 02:12:53.173014 27507 catalog_manager.cc:5484] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } }
I20240503 02:12:53.442232 28524 consensus_queue.cc:1035] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:53.542371 28815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:53.542732 28815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:53.548925 28656 consensus_queue.cc:1035] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:53.551270 28815 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:12:53.665412 28549 consensus_queue.cc:1035] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 192.905 ms
  time per row: 0.0964525 ms
I20240503 02:12:53.897670 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:39319
--local_ip_for_outbound_sockets=127.20.218.68
--tserver_master_addrs=127.20.218.126:37537
--webserver_port=45885
--webserver_interface=127.20.218.68
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:12:54.242650 28870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.242973 28870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.243099 28870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.250891 28870 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:54.251143 28870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.251331 28870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:12:54.261304 28870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:39319
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=45885
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.262801 28870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.264789 28870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.277993 28878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.278113 28875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.277993 28876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.279104 28870 server_base.cc:1034] running on GCE node
I20240503 02:12:54.279780 28870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.280777 28870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.282037 28870 hybrid_clock.cc:648] HybridClock initialized: now 1714702374281992 us; error 37 us; skew 500 ppm
I20240503 02:12:54.282397 28870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.284816 28870 webserver.cc:460] Webserver started at http://127.20.218.68:45885/ using document root <none> and password file <none>
I20240503 02:12:54.285684 28870 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.285827 28870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.292004 28870 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:12:54.295534 28884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.297094 28870 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:12:54.297310 28870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b0eb49c52e794992a8fc34173f0cdcef"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:54.298543 28870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:54.314224 28870 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:54.315083 28870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:54.315392 28870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:54.316424 28870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:54.318266 28870 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:54.318354 28870 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.318475 28870 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:54.318533 28870 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.355973 28870 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:39319
I20240503 02:12:54.356009 28996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:39319 every 8 connection(s)
I20240503 02:12:54.357748 28870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:54.368527 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 28870
I20240503 02:12:54.369398 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:32881
--local_ip_for_outbound_sockets=127.20.218.69
--tserver_master_addrs=127.20.218.126:37537
--webserver_port=38999
--webserver_interface=127.20.218.69
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:54.371896 28997 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:54.372273 28997 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.373214 28997 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:54.375025 27514 ts_manager.cc:194] Re-registered known tserver with Master: b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
I20240503 02:12:54.375917 27514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:36917
W20240503 02:12:54.719066 29001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:54.719388 29001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:54.719508 29001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:54.727525 29001 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:54.727689 29001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:54.727840 29001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:12:54.738181 29001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:32881
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=38999
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:54.739652 29001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:54.741540 29001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:54.754400 29006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.754601 29009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:54.754632 29007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:54.756366 29001 server_base.cc:1034] running on GCE node
I20240503 02:12:54.757248 29001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:54.758381 29001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:54.759620 29001 hybrid_clock.cc:648] HybridClock initialized: now 1714702374759583 us; error 36 us; skew 500 ppm
I20240503 02:12:54.759959 29001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:54.762763 29001 webserver.cc:460] Webserver started at http://127.20.218.69:38999/ using document root <none> and password file <none>
I20240503 02:12:54.763589 29001 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:54.763718 29001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:54.769721 29001 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:12:54.773267 29015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.774924 29001 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:12:54.775204 29001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a8ef897d344843e2bbbbccc342f924ca"
format_stamp: "Formatted at 2024-05-03 02:12:42 on dist-test-slave-81lq"
I20240503 02:12:54.776135 29001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:54.800768 29001 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:54.801882 29001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:54.802192 29001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:54.803313 29001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:54.805011 29001 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:54.805095 29001 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.805227 29001 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:54.805284 29001 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:54.839112 29001 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:32881
I20240503 02:12:54.839109 29127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:32881 every 8 connection(s)
I20240503 02:12:54.840749 29001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:54.850908 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 29001
I20240503 02:12:54.851542 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:37393
--local_ip_for_outbound_sockets=127.20.218.70
--tserver_master_addrs=127.20.218.126:37537
--webserver_port=45303
--webserver_interface=127.20.218.70
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:54.853860 29128 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:54.854175 29128 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:54.854957 29128 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:54.856539 27514 ts_manager.cc:194] Re-registered known tserver with Master: a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881)
I20240503 02:12:54.857789 27514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:57501
W20240503 02:12:55.202834 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:55.203158 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:55.203297 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:55.211169 29131 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:55.211320 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:55.211485 29131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:12:55.221138 29131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:37393
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=45303
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:55.222569 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:55.224403 29131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:55.237541 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.237648 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.237627 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:55.243284 29131 server_base.cc:1034] running on GCE node
I20240503 02:12:55.244019 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:55.245277 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:55.246690 29131 hybrid_clock.cc:648] HybridClock initialized: now 1714702375246554 us; error 133 us; skew 500 ppm
I20240503 02:12:55.247053 29131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:55.249972 29131 webserver.cc:460] Webserver started at http://127.20.218.70:45303/ using document root <none> and password file <none>
I20240503 02:12:55.251010 29131 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:55.251216 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:55.258956 29131 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:12:55.262881 29145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.264580 29131 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
I20240503 02:12:55.264772 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "24868c13491843548e8da6583ef1e9d5"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:55.265789 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:55.284230 29131 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:55.285120 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:55.285473 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:55.286540 29131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:55.288183 29131 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:55.288290 29131 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.288424 29131 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:55.288480 29131 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.334527 29131 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:37393
I20240503 02:12:55.334565 29257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:37393 every 8 connection(s)
I20240503 02:12:55.336064 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:55.343118 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 29131
I20240503 02:12:55.343750 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:40065
--local_ip_for_outbound_sockets=127.20.218.71
--tserver_master_addrs=127.20.218.126:37537
--webserver_port=39761
--webserver_interface=127.20.218.71
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:12:55.351301 29258 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:55.351639 29258 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:55.352450 29258 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:55.354096 27514 ts_manager.cc:194] Re-registered known tserver with Master: 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393)
I20240503 02:12:55.355356 27514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:41461
I20240503 02:12:55.379406 28997 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
W20240503 02:12:55.703593 29261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:55.703958 29261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:55.704103 29261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:55.712359 29261 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:12:55.712518 29261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:55.712684 29261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:12:55.722137 29261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:40065
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=39761
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:55.723610 29261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:55.725925 29261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:55.738688 29268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.738895 29267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:55.739699 29270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:55.739719 29261 server_base.cc:1034] running on GCE node
I20240503 02:12:55.740466 29261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:55.741513 29261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:55.742731 29261 hybrid_clock.cc:648] HybridClock initialized: now 1714702375742681 us; error 42 us; skew 500 ppm
I20240503 02:12:55.743081 29261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:55.745682 29261 webserver.cc:460] Webserver started at http://127.20.218.71:39761/ using document root <none> and password file <none>
I20240503 02:12:55.746481 29261 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:55.746608 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:55.752764 29261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.000s	sys 0.005s
I20240503 02:12:55.756122 29276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.758067 29261 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:12:55.758276 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "94278d0a47dc47f2a0d5be9589f641be"
format_stamp: "Formatted at 2024-05-03 02:12:43 on dist-test-slave-81lq"
I20240503 02:12:55.759202 29261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:55.774785 29261 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:12:55.775861 29261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:55.776237 29261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:55.777456 29261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:55.779448 29261 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:55.779536 29261 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.779670 29261 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:55.779726 29261 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:12:55.816213 29261 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:40065
I20240503 02:12:55.816232 29388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:40065 every 8 connection(s)
I20240503 02:12:55.818020 29261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:55.826802 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 29261
I20240503 02:12:55.827136 21353 test_util.cc:268] Using random seed: 1107408538
I20240503 02:12:55.835809 29389 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:12:55.836241 29389 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:55.837561 29389 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:12:55.839767 27514 ts_manager.cc:194] Re-registered known tserver with Master: 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065)
I20240503 02:12:55.841084 27514 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:46287
I20240503 02:12:55.857440 21353 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240503 02:12:55.858517 21353 test_util.cc:268] Using random seed: 1107439905
I20240503 02:12:55.860446 29128 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:12:55.879019 21353 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240503 02:12:55.879488 21353 test_util.cc:268] Using random seed: 1107460848
I20240503 02:12:55.920493 21353 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240503 02:12:55.920955 21353 test_util.cc:268] Using random seed: 1107502339
I20240503 02:12:56.001421 21353 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240503 02:12:56.030912 21353 test_util.cc:268] Using random seed: 1107612298
I20240503 02:12:56.111080 21353 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240503 02:12:56.359050 29258 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:12:56.845938 29389 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:12:57.212188 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:57.326079 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:57.350577 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:57.424388 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:57.445616 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:57.500869 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:57.508183 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:58.506703 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:58.527873 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:58.559868 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:12:58.564513 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:58.594797 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:58.595964 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:58.852823 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:59.201289 27914 consensus_queue.cc:237] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:12:59.211367 27784 raft_consensus.cc:1261] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240503 02:12:59.212371 29471 consensus_queue.cc:1035] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240503 02:12:59.217584 29526 raft_consensus.cc:2932] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:12:59.221204 27786 raft_consensus.cc:2932] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:12:59.227842 27511 catalog_manager.cc:5484] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 48, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
W20240503 02:12:59.269754 27853 consensus_peers.cc:485] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f475da5fc141450ba4ddfe739ea3ad96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.578816 27917 consensus_queue.cc:237] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:12:59.582216 27655 raft_consensus.cc:1261] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240503 02:12:59.583052 29471 consensus_queue.cc:1035] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240503 02:12:59.586283 29471 raft_consensus.cc:2932] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:12:59.590165 27655 raft_consensus.cc:2932] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:12:59.600939 27511 catalog_manager.cc:5484] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index -1 to 77, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
W20240503 02:12:59.624336 27850 consensus_peers.cc:485] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cc58692b4d144ed8bd4ad33cc8d022c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.796037 29644 ts_tablet_manager.cc:902] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Initiating tablet copy from peer eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:12:59.901391 29644 tablet_copy_client.cc:308] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45743
I20240503 02:12:59.929086 27937 tablet_copy_service.cc:140] P eaffa047cf9b43b79a5b75089ea63901: Received BeginTabletCopySession request for tablet f475da5fc141450ba4ddfe739ea3ad96 from peer 24868c13491843548e8da6583ef1e9d5 ({username='slave'} at 127.20.218.70:35311)
I20240503 02:12:59.929601 27937 tablet_copy_service.cc:161] P eaffa047cf9b43b79a5b75089ea63901: Beginning new tablet copy session on tablet f475da5fc141450ba4ddfe739ea3ad96 from peer 24868c13491843548e8da6583ef1e9d5 at {username='slave'} at 127.20.218.70:35311: session id = 24868c13491843548e8da6583ef1e9d5-f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:12:59.933840 27937 tablet_copy_source_session.cc:205] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:59.943707 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f475da5fc141450ba4ddfe739ea3ad96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.983309 29644 tablet_copy_client.cc:783] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:59.983824 29644 tablet_copy_client.cc:647] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:59.986701 29644 tablet_copy_client.cc:523] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:59.991506 29644 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Bootstrap starting.
I20240503 02:13:00.029628 29661 ts_tablet_manager.cc:902] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Initiating tablet copy from peer eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:13:00.030447 27656 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:00.044927 29644 log.cc:824] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:00.053949 27917 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 02:13:00.058161 29446 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 02:13:00.092439 29464 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:00.110554 27511 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 81, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:00.130626 27915 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:00.150923 29661 tablet_copy_client.cc:308] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45743
I20240503 02:13:00.177275 27937 tablet_copy_service.cc:140] P eaffa047cf9b43b79a5b75089ea63901: Received BeginTabletCopySession request for tablet 1cc58692b4d144ed8bd4ad33cc8d022c from peer 94278d0a47dc47f2a0d5be9589f641be ({username='slave'} at 127.20.218.71:36713)
I20240503 02:13:00.177551 27937 tablet_copy_service.cc:161] P eaffa047cf9b43b79a5b75089ea63901: Beginning new tablet copy session on tablet 1cc58692b4d144ed8bd4ad33cc8d022c from peer 94278d0a47dc47f2a0d5be9589f641be at {username='slave'} at 127.20.218.71:36713: session id = 94278d0a47dc47f2a0d5be9589f641be-1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:13:00.182498 27937 tablet_copy_source_session.cc:205] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.188366 29644 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=690 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:00.189162 29644 tablet_bootstrap.cc:492] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Bootstrap complete.
I20240503 02:13:00.189803 29644 ts_tablet_manager.cc:1367] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Time spent bootstrapping tablet: real 0.199s	user 0.124s	sys 0.016s
I20240503 02:13:00.197017 29644 raft_consensus.cc:348] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:00.197497 29661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc58692b4d144ed8bd4ad33cc8d022c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.197758 29644 raft_consensus.cc:729] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24868c13491843548e8da6583ef1e9d5, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.198709 29644 consensus_queue.cc:260] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:00.208032 29644 ts_tablet_manager.cc:1398] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.004s
I20240503 02:13:00.209424 27937 tablet_copy_service.cc:342] P eaffa047cf9b43b79a5b75089ea63901: Request end of tablet copy session 24868c13491843548e8da6583ef1e9d5-f475da5fc141450ba4ddfe739ea3ad96 received from {username='slave'} at 127.20.218.70:35311
I20240503 02:13:00.209605 27937 tablet_copy_service.cc:433] P eaffa047cf9b43b79a5b75089ea63901: ending tablet copy session 24868c13491843548e8da6583ef1e9d5-f475da5fc141450ba4ddfe739ea3ad96 on tablet f475da5fc141450ba4ddfe739ea3ad96 with peer 24868c13491843548e8da6583ef1e9d5
W20240503 02:13:00.234797 27592 consensus_peers.cc:485] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881): Couldn't send request to peer a8ef897d344843e2bbbbccc342f924ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 896ae927d34e46188120e705fb0ae361. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.236559 29661 tablet_copy_client.cc:783] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.237257 29661 tablet_copy_client.cc:647] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.239852 29661 tablet_copy_client.cc:523] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.249850 29661 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap starting.
I20240503 02:13:00.293128 29661 log.cc:824] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:00.527467 29213 raft_consensus.cc:1203] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.68]   Dedup: 1.59->[1.60-1.68]
I20240503 02:13:00.537139 27654 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:00.545675 27784 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240503 02:13:00.551605 29446 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
W20240503 02:13:00.557258 27591 consensus_peers.cc:485] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319): Couldn't send request to peer b0eb49c52e794992a8fc34173f0cdcef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a63be7e6f2da4163baced11894065444. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.568351 29446 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:00.580276 27786 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index -1 to 90, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:00.580518 27514 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 90, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:00.615152 29661 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:00.615918 29661 tablet_bootstrap.cc:492] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap complete.
I20240503 02:13:00.616549 29661 ts_tablet_manager.cc:1367] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Time spent bootstrapping tablet: real 0.367s	user 0.168s	sys 0.019s
I20240503 02:13:00.632362 29661 raft_consensus.cc:348] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:00.633224 29661 raft_consensus.cc:729] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.634207 29661 consensus_queue.cc:260] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:00.658370 29661 ts_tablet_manager.cc:1398] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be: Time spent starting tablet: real 0.042s	user 0.019s	sys 0.003s
I20240503 02:13:00.659458 27937 tablet_copy_service.cc:342] P eaffa047cf9b43b79a5b75089ea63901: Request end of tablet copy session 94278d0a47dc47f2a0d5be9589f641be-1cc58692b4d144ed8bd4ad33cc8d022c received from {username='slave'} at 127.20.218.71:36713
I20240503 02:13:00.659612 27937 tablet_copy_service.cc:433] P eaffa047cf9b43b79a5b75089ea63901: ending tablet copy session 94278d0a47dc47f2a0d5be9589f641be-1cc58692b4d144ed8bd4ad33cc8d022c on tablet 1cc58692b4d144ed8bd4ad33cc8d022c with peer 94278d0a47dc47f2a0d5be9589f641be
I20240503 02:13:00.686638 29705 ts_tablet_manager.cc:902] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:00.703161 29705 tablet_copy_client.cc:308] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:00.724260 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 896ae927d34e46188120e705fb0ae361 from peer a8ef897d344843e2bbbbccc342f924ca ({username='slave'} at 127.20.218.69:37241)
I20240503 02:13:00.724541 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 896ae927d34e46188120e705fb0ae361 from peer a8ef897d344843e2bbbbccc342f924ca at {username='slave'} at 127.20.218.69:37241: session id = a8ef897d344843e2bbbbccc342f924ca-896ae927d34e46188120e705fb0ae361
I20240503 02:13:00.728255 27676 tablet_copy_source_session.cc:205] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.737519 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896ae927d34e46188120e705fb0ae361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.758328 29705 tablet_copy_client.cc:783] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.758809 29705 tablet_copy_client.cc:647] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.762562 29705 tablet_copy_client.cc:523] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.768602 29705 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap starting.
I20240503 02:13:00.770599 27656 consensus_queue.cc:237] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:00.774670 27787 raft_consensus.cc:1261] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240503 02:13:00.777244 29262 consensus_queue.cc:1035] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240503 02:13:00.784054 29262 raft_consensus.cc:2932] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:00.795228 27784 raft_consensus.cc:2932] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:00.801436 27514 catalog_manager.cc:5484] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 81, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:00.801707 29705 log.cc:824] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Log is configured to *not* fsync() on all Append() calls
W20240503 02:13:00.821694 27589 consensus_peers.cc:485] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 376511df6d8645268134e125791f6aca. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.127704 29726 ts_tablet_manager.cc:902] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:01.165849 29344 raft_consensus.cc:1203] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.76->[1.77-1.108]   Dedup: 1.89->[1.90-1.108]
I20240503 02:13:01.301609 29705 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.302407 29705 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap complete.
I20240503 02:13:01.303098 29705 ts_tablet_manager.cc:1367] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Time spent bootstrapping tablet: real 0.535s	user 0.268s	sys 0.188s
I20240503 02:13:01.314899 29705 raft_consensus.cc:348] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:01.315573 29705 raft_consensus.cc:729] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ef897d344843e2bbbbccc342f924ca, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.317175 29705 consensus_queue.cc:260] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:01.362767 29705 ts_tablet_manager.cc:1398] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca: Time spent starting tablet: real 0.059s	user 0.026s	sys 0.024s
I20240503 02:13:01.366463 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session a8ef897d344843e2bbbbccc342f924ca-896ae927d34e46188120e705fb0ae361 received from {username='slave'} at 127.20.218.69:37241
I20240503 02:13:01.366693 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session a8ef897d344843e2bbbbccc342f924ca-896ae927d34e46188120e705fb0ae361 on tablet 896ae927d34e46188120e705fb0ae361 with peer a8ef897d344843e2bbbbccc342f924ca
I20240503 02:13:01.405748 29733 ts_tablet_manager.cc:902] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:01.439253 29726 tablet_copy_client.cc:308] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:01.493791 29733 tablet_copy_client.cc:308] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:01.542352 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet a63be7e6f2da4163baced11894065444 from peer b0eb49c52e794992a8fc34173f0cdcef ({username='slave'} at 127.20.218.68:48001)
I20240503 02:13:01.542559 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet a63be7e6f2da4163baced11894065444 from peer b0eb49c52e794992a8fc34173f0cdcef at {username='slave'} at 127.20.218.68:48001: session id = b0eb49c52e794992a8fc34173f0cdcef-a63be7e6f2da4163baced11894065444
I20240503 02:13:01.545938 27676 tablet_copy_source_session.cc:205] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.556180 29726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63be7e6f2da4163baced11894065444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.587278 29726 tablet_copy_client.cc:783] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.588178 29726 tablet_copy_client.cc:647] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.591240 29726 tablet_copy_client.cc:523] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.595189 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 376511df6d8645268134e125791f6aca from peer 94278d0a47dc47f2a0d5be9589f641be ({username='slave'} at 127.20.218.71:52495)
I20240503 02:13:01.595546 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 376511df6d8645268134e125791f6aca from peer 94278d0a47dc47f2a0d5be9589f641be at {username='slave'} at 127.20.218.71:52495: session id = 94278d0a47dc47f2a0d5be9589f641be-376511df6d8645268134e125791f6aca
I20240503 02:13:01.595978 29726 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap starting.
I20240503 02:13:01.604274 27676 tablet_copy_source_session.cc:205] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.629729 29733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376511df6d8645268134e125791f6aca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.630153 29726 log.cc:824] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Log is configured to *not* fsync() on all Append() calls
I20240503 02:13:01.656021 29733 tablet_copy_client.cc:783] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.656661 29733 tablet_copy_client.cc:647] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.679517 29733 tablet_copy_client.cc:523] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.684552 29733 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap starting.
I20240503 02:13:01.714260 29083 raft_consensus.cc:1203] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.106]   Dedup: 1.90->[1.91-1.106]
I20240503 02:13:01.897070 29726 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=109 overwritten=0 applied=109 ignored=0} inserts{seen=781 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.899120 29726 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap complete.
I20240503 02:13:01.899791 29726 ts_tablet_manager.cc:1367] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Time spent bootstrapping tablet: real 0.304s	user 0.174s	sys 0.033s
I20240503 02:13:01.911790 27785 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:01.915148 29726 raft_consensus.cc:348] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:01.915843 29726 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.917037 29726 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 109, Last appended: 1.109, Last appended by leader: 109, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:01.931143 29726 ts_tablet_manager.cc:1398] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef: Time spent starting tablet: real 0.031s	user 0.014s	sys 0.004s
I20240503 02:13:01.932147 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session b0eb49c52e794992a8fc34173f0cdcef-a63be7e6f2da4163baced11894065444 received from {username='slave'} at 127.20.218.68:48001
I20240503 02:13:01.932348 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session b0eb49c52e794992a8fc34173f0cdcef-a63be7e6f2da4163baced11894065444 on tablet a63be7e6f2da4163baced11894065444 with peer b0eb49c52e794992a8fc34173f0cdcef
I20240503 02:13:01.939041 27916 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 88. (index mismatch)
I20240503 02:13:01.940090 29423 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20240503 02:13:01.952157 29423 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:01.962898 27917 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:01.968483 27514 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 88, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:01.981094 28952 raft_consensus.cc:1203] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.89->[1.90-1.114]   Dedup: 1.109->[1.110-1.114]
I20240503 02:13:02.082090 29733 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:02.082684 29733 tablet_bootstrap.cc:492] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap complete.
I20240503 02:13:02.083076 29733 ts_tablet_manager.cc:1367] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Time spent bootstrapping tablet: real 0.399s	user 0.188s	sys 0.047s
I20240503 02:13:02.085214 29733 raft_consensus.cc:348] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:02.095736 29733 raft_consensus.cc:729] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.110041 29733 consensus_queue.cc:260] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:02.131532 29733 ts_tablet_manager.cc:1398] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be: Time spent starting tablet: real 0.048s	user 0.005s	sys 0.000s
I20240503 02:13:02.144477 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 94278d0a47dc47f2a0d5be9589f641be-376511df6d8645268134e125791f6aca received from {username='slave'} at 127.20.218.71:52495
I20240503 02:13:02.144650 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 94278d0a47dc47f2a0d5be9589f641be-376511df6d8645268134e125791f6aca on tablet 376511df6d8645268134e125791f6aca with peer 94278d0a47dc47f2a0d5be9589f641be
W20240503 02:13:02.156870 27723 consensus_peers.cc:485] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ec65a7b8d0943119b929ae3a9544ec0. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:02.174341 27850 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:40065, user_credentials={real_user=slave}} blocked reactor thread for 108091us
I20240503 02:13:02.248788 27784 consensus_queue.cc:237] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:02.275939 27656 raft_consensus.cc:1261] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20240503 02:13:02.278681 29607 consensus_queue.cc:1035] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20240503 02:13:02.291465 29441 raft_consensus.cc:2932] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index -1 to 116, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:02.294441 27656 raft_consensus.cc:2932] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index -1 to 116, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:02.302922 27514 catalog_manager.cc:5484] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 116, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
W20240503 02:13:02.316114 27723 consensus_peers.cc:485] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 -> Peer a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881): Couldn't send request to peer a8ef897d344843e2bbbbccc342f924ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a52a6040dd89493da32af4b283774f03. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.455685 29344 raft_consensus.cc:1203] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.80->[1.81-1.107]   Dedup: 1.97->[1.98-1.107]
I20240503 02:13:02.474486 29824 ts_tablet_manager.cc:902] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:13:02.507706 29824 tablet_copy_client.cc:308] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:13:02.596746 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet 4ec65a7b8d0943119b929ae3a9544ec0 from peer 24868c13491843548e8da6583ef1e9d5 ({username='slave'} at 127.20.218.70:38931)
I20240503 02:13:02.597066 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet 4ec65a7b8d0943119b929ae3a9544ec0 from peer 24868c13491843548e8da6583ef1e9d5 at {username='slave'} at 127.20.218.70:38931: session id = 24868c13491843548e8da6583ef1e9d5-4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:13:02.607926 27807 tablet_copy_source_session.cc:205] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:02.614874 29824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec65a7b8d0943119b929ae3a9544ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.622023 29824 tablet_copy_client.cc:783] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:02.622467 29824 tablet_copy_client.cc:647] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:02.642941 29824 tablet_copy_client.cc:523] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:02.657685 29824 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap starting.
I20240503 02:13:02.893672 27784 consensus_queue.cc:237] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:02.898802 27917 raft_consensus.cc:1261] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240503 02:13:02.906383 29607 consensus_queue.cc:1035] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240503 02:13:02.910341 29607 raft_consensus.cc:2932] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:02.917467 27914 raft_consensus.cc:2932] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index -1 to 123, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:02.922098 27509 catalog_manager.cc:5484] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 123, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:02.933367 29847 ts_tablet_manager.cc:902] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
W20240503 02:13:02.941098 27722 consensus_peers.cc:485] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 -> Peer b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319): Couldn't send request to peer b0eb49c52e794992a8fc34173f0cdcef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bf2a0fe803b42e4a706f50762a4b87d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.964258 29847 tablet_copy_client.cc:308] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:13:03.009392 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet a52a6040dd89493da32af4b283774f03 from peer a8ef897d344843e2bbbbccc342f924ca ({username='slave'} at 127.20.218.69:39757)
I20240503 02:13:03.009678 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet a52a6040dd89493da32af4b283774f03 from peer a8ef897d344843e2bbbbccc342f924ca at {username='slave'} at 127.20.218.69:39757: session id = a8ef897d344843e2bbbbccc342f924ca-a52a6040dd89493da32af4b283774f03
I20240503 02:13:03.014000 27807 tablet_copy_source_session.cc:205] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.017885 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52a6040dd89493da32af4b283774f03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.049188 29824 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=0 applied=96 ignored=0} inserts{seen=727 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.049824 29824 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap complete.
I20240503 02:13:03.050168 29824 ts_tablet_manager.cc:1367] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Time spent bootstrapping tablet: real 0.395s	user 0.190s	sys 0.025s
I20240503 02:13:03.054396 29824 raft_consensus.cc:348] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:03.054649 29824 raft_consensus.cc:729] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24868c13491843548e8da6583ef1e9d5, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.054922 29824 consensus_queue.cc:260] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 1.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:03.058207 29847 tablet_copy_client.cc:783] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.058589 29847 tablet_copy_client.cc:647] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.062189 29847 tablet_copy_client.cc:523] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.079056 29847 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap starting.
I20240503 02:13:03.082639 29824 ts_tablet_manager.cc:1398] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5: Time spent starting tablet: real 0.029s	user 0.013s	sys 0.006s
I20240503 02:13:03.084295 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session 24868c13491843548e8da6583ef1e9d5-4ec65a7b8d0943119b929ae3a9544ec0 received from {username='slave'} at 127.20.218.70:38931
I20240503 02:13:03.084483 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session 24868c13491843548e8da6583ef1e9d5-4ec65a7b8d0943119b929ae3a9544ec0 on tablet 4ec65a7b8d0943119b929ae3a9544ec0 with peer 24868c13491843548e8da6583ef1e9d5
I20240503 02:13:03.270349 27656 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:03.289583 27786 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
W20240503 02:13:03.296038 27589 consensus_peers.cc:485] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 878e847516c74149966db83f6f70c6d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.296352 29262 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240503 02:13:03.303500 29262 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:03.308264 27786 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index -1 to 95, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:03.316848 27514 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 95, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:03.411633 29880 ts_tablet_manager.cc:902] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:13:03.510068 29880 tablet_copy_client.cc:308] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:13:03.544899 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet 8bf2a0fe803b42e4a706f50762a4b87d from peer b0eb49c52e794992a8fc34173f0cdcef ({username='slave'} at 127.20.218.68:58065)
I20240503 02:13:03.545106 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet 8bf2a0fe803b42e4a706f50762a4b87d from peer b0eb49c52e794992a8fc34173f0cdcef at {username='slave'} at 127.20.218.68:58065: session id = b0eb49c52e794992a8fc34173f0cdcef-8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:13:03.551807 27807 tablet_copy_source_session.cc:205] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.553857 29880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf2a0fe803b42e4a706f50762a4b87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.555167 29847 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=128 overwritten=0 applied=128 ignored=0} inserts{seen=800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.555958 29847 tablet_bootstrap.cc:492] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap complete.
I20240503 02:13:03.556378 29847 ts_tablet_manager.cc:1367] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Time spent bootstrapping tablet: real 0.477s	user 0.204s	sys 0.016s
I20240503 02:13:03.557610 29847 raft_consensus.cc:348] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:03.558038 29847 raft_consensus.cc:729] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ef897d344843e2bbbbccc342f924ca, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.558408 29847 consensus_queue.cc:260] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 128, Last appended: 1.128, Last appended by leader: 128, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:03.562918 29880 tablet_copy_client.cc:783] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.563274 29880 tablet_copy_client.cc:647] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.570698 29847 ts_tablet_manager.cc:1398] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20240503 02:13:03.586346 27806 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session a8ef897d344843e2bbbbccc342f924ca-a52a6040dd89493da32af4b283774f03 received from {username='slave'} at 127.20.218.69:39757
I20240503 02:13:03.586531 27806 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session a8ef897d344843e2bbbbccc342f924ca-a52a6040dd89493da32af4b283774f03 on tablet a52a6040dd89493da32af4b283774f03 with peer a8ef897d344843e2bbbbccc342f924ca
I20240503 02:13:03.600096 29880 tablet_copy_client.cc:523] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.604945 29880 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap starting.
I20240503 02:13:03.635854 29212 raft_consensus.cc:1203] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.110]   Dedup: 1.96->[1.97-1.110]
I20240503 02:13:03.700937 29083 raft_consensus.cc:1203] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.115->[1.116-1.137]   Dedup: 1.128->[1.129-1.137]
I20240503 02:13:03.735935 27786 consensus_queue.cc:237] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
W20240503 02:13:03.738684 27723 consensus_peers.cc:485] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 -> Peer a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881): Couldn't send request to peer a8ef897d344843e2bbbbccc342f924ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddd409a1c4554665831eceef070ab660. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.741531 27655 raft_consensus.cc:1261] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240503 02:13:03.749753 29609 consensus_queue.cc:1035] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240503 02:13:03.766372 29423 raft_consensus.cc:2932] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:03.773257 27514 catalog_manager.cc:5484] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 113, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:03.778209 27655 raft_consensus.cc:2932] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:03.811658 29910 ts_tablet_manager.cc:902] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:03.949532 29910 tablet_copy_client.cc:308] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:03.964064 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 878e847516c74149966db83f6f70c6d0 from peer 94278d0a47dc47f2a0d5be9589f641be ({username='slave'} at 127.20.218.71:52495)
I20240503 02:13:03.964293 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 878e847516c74149966db83f6f70c6d0 from peer 94278d0a47dc47f2a0d5be9589f641be at {username='slave'} at 127.20.218.71:52495: session id = 94278d0a47dc47f2a0d5be9589f641be-878e847516c74149966db83f6f70c6d0
I20240503 02:13:03.967479 27676 tablet_copy_source_session.cc:205] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.004393 29880 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=128 overwritten=0 applied=128 ignored=0} inserts{seen=759 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.005074 29880 tablet_bootstrap.cc:492] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap complete.
I20240503 02:13:04.005525 29880 ts_tablet_manager.cc:1367] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Time spent bootstrapping tablet: real 0.401s	user 0.197s	sys 0.024s
I20240503 02:13:04.007571 29880 raft_consensus.cc:348] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:04.007983 29880 raft_consensus.cc:729] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.008307 29880 consensus_queue.cc:260] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 128, Last appended: 1.128, Last appended by leader: 128, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:04.017854 29880 ts_tablet_manager.cc:1398] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.005s
I20240503 02:13:04.020203 29910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878e847516c74149966db83f6f70c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.027590 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session b0eb49c52e794992a8fc34173f0cdcef-8bf2a0fe803b42e4a706f50762a4b87d received from {username='slave'} at 127.20.218.68:58065
I20240503 02:13:04.027774 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session b0eb49c52e794992a8fc34173f0cdcef-8bf2a0fe803b42e4a706f50762a4b87d on tablet 8bf2a0fe803b42e4a706f50762a4b87d with peer b0eb49c52e794992a8fc34173f0cdcef
I20240503 02:13:04.030247 29910 tablet_copy_client.cc:783] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.030886 29910 tablet_copy_client.cc:647] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.036681 29910 tablet_copy_client.cc:523] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:04.049960 29910 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap starting.
I20240503 02:13:04.145453 29927 ts_tablet_manager.cc:902] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:13:04.233053 29927 tablet_copy_client.cc:308] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:13:04.243469 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet ddd409a1c4554665831eceef070ab660 from peer a8ef897d344843e2bbbbccc342f924ca ({username='slave'} at 127.20.218.69:39757)
I20240503 02:13:04.243692 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet ddd409a1c4554665831eceef070ab660 from peer a8ef897d344843e2bbbbccc342f924ca at {username='slave'} at 127.20.218.69:39757: session id = a8ef897d344843e2bbbbccc342f924ca-ddd409a1c4554665831eceef070ab660
I20240503 02:13:04.246605 27807 tablet_copy_source_session.cc:205] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.248378 29927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd409a1c4554665831eceef070ab660. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.258309 29927 tablet_copy_client.cc:783] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.258921 29927 tablet_copy_client.cc:647] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.263437 29927 tablet_copy_client.cc:523] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:04.268679 29927 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap starting.
W20240503 02:13:04.295117 27592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 87755us
I20240503 02:13:04.315929 27917 consensus_queue.cc:237] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:04.374544 29910 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=103 ignored=0} inserts{seen=788 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.375475 29910 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap complete.
I20240503 02:13:04.376093 29910 ts_tablet_manager.cc:1367] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Time spent bootstrapping tablet: real 0.326s	user 0.162s	sys 0.048s
I20240503 02:13:04.378044 29910 raft_consensus.cc:348] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:04.380060 29910 raft_consensus.cc:729] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.380561 29910 consensus_queue.cc:260] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:04.382130 28952 raft_consensus.cc:1203] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.122->[1.123-1.134]   Dedup: 1.128->[1.129-1.134]
I20240503 02:13:04.406461 29910 ts_tablet_manager.cc:1398] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Time spent starting tablet: real 0.030s	user 0.000s	sys 0.007s
I20240503 02:13:04.413504 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 94278d0a47dc47f2a0d5be9589f641be-878e847516c74149966db83f6f70c6d0 received from {username='slave'} at 127.20.218.71:52495
I20240503 02:13:04.413681 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 94278d0a47dc47f2a0d5be9589f641be-878e847516c74149966db83f6f70c6d0 on tablet 878e847516c74149966db83f6f70c6d0 with peer 94278d0a47dc47f2a0d5be9589f641be
I20240503 02:13:04.437487 27784 raft_consensus.cc:1261] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:04.438305 29796 consensus_queue.cc:1035] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:04.473816 27852 consensus_peers.cc:485] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 -> Peer b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319): Couldn't send request to peer b0eb49c52e794992a8fc34173f0cdcef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42ad89cd14fb4dcfbfa7caeb944549c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:04.487473 29951 raft_consensus.cc:2932] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:04.500612 27784 raft_consensus.cc:2932] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:04.510521 27514 catalog_manager.cc:5484] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 2, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:04.556883 29927 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=121 overwritten=0 applied=121 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.557487 29927 tablet_bootstrap.cc:492] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap complete.
I20240503 02:13:04.557888 29927 ts_tablet_manager.cc:1367] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Time spent bootstrapping tablet: real 0.289s	user 0.186s	sys 0.025s
I20240503 02:13:04.558833 29927 raft_consensus.cc:348] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:04.559176 29927 raft_consensus.cc:729] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ef897d344843e2bbbbccc342f924ca, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.559532 29927 consensus_queue.cc:260] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 121, Last appended: 1.121, Last appended by leader: 121, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:04.614275 29927 ts_tablet_manager.cc:1398] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca: Time spent starting tablet: real 0.056s	user 0.032s	sys 0.023s
I20240503 02:13:04.618353 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session a8ef897d344843e2bbbbccc342f924ca-ddd409a1c4554665831eceef070ab660 received from {username='slave'} at 127.20.218.69:39757
I20240503 02:13:04.618523 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session a8ef897d344843e2bbbbccc342f924ca-ddd409a1c4554665831eceef070ab660 on tablet ddd409a1c4554665831eceef070ab660 with peer a8ef897d344843e2bbbbccc342f924ca
I20240503 02:13:04.622789 29082 raft_consensus.cc:1203] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.112->[1.113-1.123]   Dedup: 1.121->[1.122-1.123]
I20240503 02:13:04.728561 29344 raft_consensus.cc:1203] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.114]   Dedup: 1.103->[1.104-1.114]
I20240503 02:13:04.911424 29977 ts_tablet_manager.cc:902] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Initiating tablet copy from peer eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:13:05.064040 29977 tablet_copy_client.cc:308] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45743
I20240503 02:13:05.117857 27654 consensus_queue.cc:237] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:05.126123 27785 raft_consensus.cc:1261] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 109. (index mismatch)
W20240503 02:13:05.133208 27589 consensus_peers.cc:485] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6af88acfe24e49279a98b5dcf417bed7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:05.133466 27937 tablet_copy_service.cc:140] P eaffa047cf9b43b79a5b75089ea63901: Received BeginTabletCopySession request for tablet 42ad89cd14fb4dcfbfa7caeb944549c2 from peer b0eb49c52e794992a8fc34173f0cdcef ({username='slave'} at 127.20.218.68:54841)
I20240503 02:13:05.133641 27937 tablet_copy_service.cc:161] P eaffa047cf9b43b79a5b75089ea63901: Beginning new tablet copy session on tablet 42ad89cd14fb4dcfbfa7caeb944549c2 from peer b0eb49c52e794992a8fc34173f0cdcef at {username='slave'} at 127.20.218.68:54841: session id = b0eb49c52e794992a8fc34173f0cdcef-42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:13:05.135738 29946 consensus_queue.cc:1035] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 106, Time since last communication: 0.000s
I20240503 02:13:05.136582 27937 tablet_copy_source_session.cc:205] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:05.138900 29977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ad89cd14fb4dcfbfa7caeb944549c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.152415 29977 tablet_copy_client.cc:783] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:05.152212 29793 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index -1 to 109, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:05.152755 29977 tablet_copy_client.cc:647] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:05.173421 27784 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index -1 to 109, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:05.175496 29977 tablet_copy_client.cc:523] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:05.175352 27514 catalog_manager.cc:5484] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 109, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:05.185855 29977 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap starting.
I20240503 02:13:05.237107 29977 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:05.237635 29977 tablet_bootstrap.cc:492] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap complete.
I20240503 02:13:05.237901 29977 ts_tablet_manager.cc:1367] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Time spent bootstrapping tablet: real 0.052s	user 0.009s	sys 0.001s
I20240503 02:13:05.238806 29977 raft_consensus.cc:348] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:05.239712 29977 raft_consensus.cc:729] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.246685 29977 consensus_queue.cc:260] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:05.309999 29977 ts_tablet_manager.cc:1398] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef: Time spent starting tablet: real 0.072s	user 0.015s	sys 0.013s
I20240503 02:13:05.311100 27937 tablet_copy_service.cc:342] P eaffa047cf9b43b79a5b75089ea63901: Request end of tablet copy session b0eb49c52e794992a8fc34173f0cdcef-42ad89cd14fb4dcfbfa7caeb944549c2 received from {username='slave'} at 127.20.218.68:54841
I20240503 02:13:05.311276 27937 tablet_copy_service.cc:433] P eaffa047cf9b43b79a5b75089ea63901: ending tablet copy session b0eb49c52e794992a8fc34173f0cdcef-42ad89cd14fb4dcfbfa7caeb944549c2 on tablet 42ad89cd14fb4dcfbfa7caeb944549c2 with peer b0eb49c52e794992a8fc34173f0cdcef
I20240503 02:13:05.355899 28951 raft_consensus.cc:1203] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240503 02:13:05.626504 27656 consensus_queue.cc:237] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 140, Committed index: 140, Last appended: 1.140, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:05.632112 27914 raft_consensus.cc:1261] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 140. Preceding OpId from leader: term: 1 index: 141. (index mismatch)
I20240503 02:13:05.632874 29946 consensus_queue.cc:1035] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 141, Last known committed idx: 140, Time since last communication: 0.000s
W20240503 02:13:05.635885 27592 consensus_peers.cc:485] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer a8ef897d344843e2bbbbccc342f924ca (127.20.218.69:32881): Couldn't send request to peer a8ef897d344843e2bbbbccc342f924ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b842b202e3b4d17afd245f0d90a34a4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:05.636428 29946 raft_consensus.cc:2932] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.141: config changed from index -1 to 141, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:05.642211 27509 catalog_manager.cc:5484] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index -1 to 141, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:05.644275 27914 raft_consensus.cc:2932] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.141: config changed from index -1 to 141, VOTER a8ef897d344843e2bbbbccc342f924ca (127.20.218.69) added. New config: { opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:05.763958 30024 ts_tablet_manager.cc:902] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:05.982795 27655 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
W20240503 02:13:05.986614 27591 consensus_peers.cc:485] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319): Couldn't send request to peer b0eb49c52e794992a8fc34173f0cdcef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4395a743162549ca96757c5838c0e190. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.101920 30024 tablet_copy_client.cc:308] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:06.102952 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 6af88acfe24e49279a98b5dcf417bed7 from peer 94278d0a47dc47f2a0d5be9589f641be ({username='slave'} at 127.20.218.71:52495)
I20240503 02:13:06.103180 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 6af88acfe24e49279a98b5dcf417bed7 from peer 94278d0a47dc47f2a0d5be9589f641be at {username='slave'} at 127.20.218.71:52495: session id = 94278d0a47dc47f2a0d5be9589f641be-6af88acfe24e49279a98b5dcf417bed7
I20240503 02:13:06.106627 27676 tablet_copy_source_session.cc:205] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.108268 30024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af88acfe24e49279a98b5dcf417bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.118534 30024 tablet_copy_client.cc:783] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:06.118930 30024 tablet_copy_client.cc:647] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:06.135926 27917 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch)
I20240503 02:13:06.139075 30024 tablet_copy_client.cc:523] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.139056 29871 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 124, Time since last communication: 0.000s
I20240503 02:13:06.150669 30024 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap starting.
I20240503 02:13:06.155164 29793 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index -1 to 126, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:06.165185 27915 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index -1 to 126, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:06.171994 27514 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index -1 to 126, VOTER b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:06.196372 30043 ts_tablet_manager.cc:902] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:06.418749 30043 tablet_copy_client.cc:308] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:06.421926 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 6b842b202e3b4d17afd245f0d90a34a4 from peer a8ef897d344843e2bbbbccc342f924ca ({username='slave'} at 127.20.218.69:37241)
I20240503 02:13:06.422156 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 6b842b202e3b4d17afd245f0d90a34a4 from peer a8ef897d344843e2bbbbccc342f924ca at {username='slave'} at 127.20.218.69:37241: session id = a8ef897d344843e2bbbbccc342f924ca-6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:13:06.425014 27676 tablet_copy_source_session.cc:205] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.437940 30043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b842b202e3b4d17afd245f0d90a34a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.444731 30043 tablet_copy_client.cc:783] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:06.445259 30043 tablet_copy_client.cc:647] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:06.465147 30043 tablet_copy_client.cc:523] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.469110 30043 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap starting.
I20240503 02:13:06.626550 30066 ts_tablet_manager.cc:902] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:06.734428 30024 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap replayed 1/1 log segments. Stats: ops{read=119 overwritten=0 applied=119 ignored=0} inserts{seen=804 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.735096 30024 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap complete.
I20240503 02:13:06.735508 30024 ts_tablet_manager.cc:1367] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Time spent bootstrapping tablet: real 0.585s	user 0.245s	sys 0.045s
I20240503 02:13:06.737082 30024 raft_consensus.cc:348] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:06.737562 30024 raft_consensus.cc:729] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.737991 30024 consensus_queue.cc:260] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 119, Last appended: 1.119, Last appended by leader: 119, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:06.758757 30024 ts_tablet_manager.cc:1398] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Time spent starting tablet: real 0.023s	user 0.021s	sys 0.000s
I20240503 02:13:06.766814 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 94278d0a47dc47f2a0d5be9589f641be-6af88acfe24e49279a98b5dcf417bed7 received from {username='slave'} at 127.20.218.71:52495
I20240503 02:13:06.767010 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 94278d0a47dc47f2a0d5be9589f641be-6af88acfe24e49279a98b5dcf417bed7 on tablet 6af88acfe24e49279a98b5dcf417bed7 with peer 94278d0a47dc47f2a0d5be9589f641be
I20240503 02:13:06.799554 29343 raft_consensus.cc:1203] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.108->[1.109-1.125]   Dedup: 1.119->[1.120-1.125]
I20240503 02:13:07.127810 30066 tablet_copy_client.cc:308] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:07.129474 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 4395a743162549ca96757c5838c0e190 from peer b0eb49c52e794992a8fc34173f0cdcef ({username='slave'} at 127.20.218.68:48001)
I20240503 02:13:07.129671 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 4395a743162549ca96757c5838c0e190 from peer b0eb49c52e794992a8fc34173f0cdcef at {username='slave'} at 127.20.218.68:48001: session id = b0eb49c52e794992a8fc34173f0cdcef-4395a743162549ca96757c5838c0e190
I20240503 02:13:07.134406 27676 tablet_copy_source_session.cc:205] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:07.143615 30066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395a743162549ca96757c5838c0e190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.153846 30066 tablet_copy_client.cc:783] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.154978 30066 tablet_copy_client.cc:647] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.158948 30066 tablet_copy_client.cc:523] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.167810 30066 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap starting.
I20240503 02:13:07.246855 30043 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=146 overwritten=0 applied=146 ignored=0} inserts{seen=831 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.247368 30043 tablet_bootstrap.cc:492] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Bootstrap complete.
I20240503 02:13:07.247694 30043 ts_tablet_manager.cc:1367] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Time spent bootstrapping tablet: real 0.779s	user 0.259s	sys 0.020s
I20240503 02:13:07.248606 30043 raft_consensus.cc:348] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:07.253948 30043 raft_consensus.cc:729] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ef897d344843e2bbbbccc342f924ca, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.257659 30043 consensus_queue.cc:260] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 141 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:07.274657 30043 ts_tablet_manager.cc:1398] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20240503 02:13:07.289161 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session a8ef897d344843e2bbbbccc342f924ca-6b842b202e3b4d17afd245f0d90a34a4 received from {username='slave'} at 127.20.218.69:37241
I20240503 02:13:07.289382 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session a8ef897d344843e2bbbbccc342f924ca-6b842b202e3b4d17afd245f0d90a34a4 on tablet 6b842b202e3b4d17afd245f0d90a34a4 with peer a8ef897d344843e2bbbbccc342f924ca
I20240503 02:13:07.523165 29082 raft_consensus.cc:1203] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.140->[1.141-1.160]   Dedup: 1.146->[1.147-1.160]
I20240503 02:13:07.771904 30066 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=136 ignored=0} inserts{seen=808 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.772548 30066 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Bootstrap complete.
I20240503 02:13:07.772970 30066 ts_tablet_manager.cc:1367] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Time spent bootstrapping tablet: real 0.605s	user 0.210s	sys 0.034s
I20240503 02:13:07.776420 30066 raft_consensus.cc:348] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:07.776841 30066 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.777491 30066 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 136, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:07.886415 30066 ts_tablet_manager.cc:1398] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef: Time spent starting tablet: real 0.113s	user 0.019s	sys 0.012s
I20240503 02:13:07.889079 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session b0eb49c52e794992a8fc34173f0cdcef-4395a743162549ca96757c5838c0e190 received from {username='slave'} at 127.20.218.68:48001
I20240503 02:13:07.889261 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session b0eb49c52e794992a8fc34173f0cdcef-4395a743162549ca96757c5838c0e190 on tablet 4395a743162549ca96757c5838c0e190 with peer b0eb49c52e794992a8fc34173f0cdcef
I20240503 02:13:07.994925 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:08.004855 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:08.022014 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:08.049525 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:08.121151 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:08.129230 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:08.147444 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:08.273471 28951 raft_consensus.cc:1203] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.125->[1.126-1.148]   Dedup: 1.136->[1.137-1.148]
I20240503 02:13:09.941506 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:13:09.943472 27852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:44115, user_credentials={real_user=slave}} blocked reactor thread for 110151us
I20240503 02:13:09.970086 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:09.980015 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:09.999222 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:10.059682 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:10.079808 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:10.171344 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:11.611429 27591 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:39319, user_credentials={real_user=slave}} blocked reactor thread for 93256.7us
I20240503 02:13:12.101439 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:12.131783 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:12.143281 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:12.156118 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:12.192780 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:12.254703 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:12.267602 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:13:12.884579 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 57549.7us
I20240503 02:13:13.035028 27653 consensus_queue.cc:237] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:13.055279 29341 raft_consensus.cc:1261] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240503 02:13:13.076936 30170 consensus_queue.cc:1035] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 205, Time since last communication: 0.000s
I20240503 02:13:13.092029 30127 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 109 to 207, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:13.096423 29344 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 109 to 207, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:13.104871 27514 catalog_manager.cc:5484] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 109 to 207, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:13.210188 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af88acfe24e49279a98b5dcf417bed7 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:51446
I20240503 02:13:13.233891 30309 tablet_replica.cc:330] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:13:13.234619 30309 raft_consensus.cc:2229] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.235018 30309 pending_rounds.cc:62] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Trying to abort 1 pending ops.
I20240503 02:13:13.235196 30309 pending_rounds.cc:68] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Aborting op as it isn't in flight: id { term: 1 index: 206 } timestamp: 7023421001743400960 op_type: WRITE_OP write_request { tablet_id: "6af88acfe24e49279a98b5dcf417bed7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\350\353\342W\277\032\222\345\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""48d05b18ce195401" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023421001588486144 authz_token { token_data: "\010\324\222\321\261\006\"3\n\005slave\022*\n 1397d007101b4cf6a7622c4a8aa5841c\020\001\030\001 \001(\001" signature: "<redacted>""+Q\323*J\355K\326\346\310\226\024\023nj\302|\013\363\013D\341v\265tO\344b\222Vd)\304J]Ik\275\346\272H\033\177\022\236\355\246\267\222\347\026\303\324\327\260\247\330\310\251\010\221\302\010\232" signing_key_seq_num: 0 } } request_id { client_id: "6fd88b80ccca41b2962b3d41716c3802" seq_no: 635 first_incomplete_seq_no: 635 attempt_no: 0 }
I20240503 02:13:13.256357 30309 raft_consensus.cc:2258] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.278604 30309 ts_tablet_manager.cc:1875] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:13.292797 30309 ts_tablet_manager.cc:1888] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.206
I20240503 02:13:13.293103 30309 log.cc:1196] T 6af88acfe24e49279a98b5dcf417bed7 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/6af88acfe24e49279a98b5dcf417bed7
I20240503 02:13:13.295199 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet 6af88acfe24e49279a98b5dcf417bed7 (table rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]) successfully deleted
I20240503 02:13:14.105934 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.200227 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.229014 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.271533 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:13:14.439635 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:14.524276 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:14.597615 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:14.910063 27914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2"
dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43106
I20240503 02:13:14.910367 27914 raft_consensus.cc:595] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:14.962617 30319 raft_consensus.cc:979] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: : Instructing follower acd475a9eee143a89c1d28a05fa51d26 to start an election
I20240503 02:13:14.962811 30319 raft_consensus.cc:1067] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Signalling peer acd475a9eee143a89c1d28a05fa51d26 to start an election
I20240503 02:13:14.964494 27786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2"
dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
 from {username='slave'} at 127.20.218.67:47269
I20240503 02:13:14.964803 27786 raft_consensus.cc:482] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:14.964923 27786 raft_consensus.cc:3037] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.967842 27786 raft_consensus.cc:504] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:14.984575 27914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:13:14.984853 27914 raft_consensus.cc:3032] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:14.984973 27914 raft_consensus.cc:729] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:13:14.985302 27914 consensus_queue.cc:260] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:14.985838 27914 raft_consensus.cc:3037] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:14.989096 27914 raft_consensus.cc:2454] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 2.
I20240503 02:13:14.990271 27723 leader_election.cc:302] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:13:14.990661 30029 raft_consensus.cc:2781] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:14.990813 30029 raft_consensus.cc:686] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:13:14.991137 30029 consensus_queue.cc:237] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:14.993294 27786 leader_election.cc:288] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 2 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743), b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
I20240503 02:13:15.004864 28950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ad89cd14fb4dcfbfa7caeb944549c2" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b0eb49c52e794992a8fc34173f0cdcef"
I20240503 02:13:15.007755 28950 raft_consensus.cc:3037] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:15.127064 28950 raft_consensus.cc:2454] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 2.
I20240503 02:13:15.196135 27514 catalog_manager.cc:5484] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 1 to 2, leader changed from eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 2 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:15.562531 28951 raft_consensus.cc:1261] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:15.569262 30142 consensus_queue.cc:1035] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:15.614965 27917 raft_consensus.cc:1261] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:15.661703 30385 consensus_queue.cc:1035] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:16.599601 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.613098 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:16.619446 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.633674 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:16.634522 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:13:16.639845 27589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:46503, user_credentials={real_user=slave}} blocked reactor thread for 45031.2us
I20240503 02:13:16.750967 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:16.756531 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:17.569461 27787 consensus_queue.cc:237] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:17.595292 28952 raft_consensus.cc:1261] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:17.597263 30386 consensus_queue.cc:1035] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:17.616326 30414 raft_consensus.cc:2932] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:17.623234 27514 catalog_manager.cc:5484] T 42ad89cd14fb4dcfbfa7caeb944549c2 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 2 to 4, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:17.633023 28951 raft_consensus.cc:2932] T 42ad89cd14fb4dcfbfa7caeb944549c2 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:18.090975 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ad89cd14fb4dcfbfa7caeb944549c2 with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40704
I20240503 02:13:18.186758 30509 tablet_replica.cc:330] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:13:18.187325 30509 raft_consensus.cc:2229] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.187624 30509 raft_consensus.cc:2258] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.189559 30509 ts_tablet_manager.cc:1875] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:18.202584 30509 ts_tablet_manager.cc:1888] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:13:18.202792 30509 log.cc:1196] T 42ad89cd14fb4dcfbfa7caeb944549c2 P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:13:18.209614 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 42ad89cd14fb4dcfbfa7caeb944549c2 (table TestTable [id=c3ebe602e0a946b19616e7756eaf7161]) successfully deleted
W20240503 02:13:18.617947 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 60918.3us
I20240503 02:13:19.224026 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.236729 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.258306 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.310766 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:19.408453 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:19.472304 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:19.585078 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:13:19.849632 27852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:44115, user_credentials={real_user=slave}} blocked reactor thread for 59179.4us
I20240503 02:13:20.510304 27784 consensus_queue.cc:237] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:20.533272 29080 raft_consensus.cc:1261] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240503 02:13:20.535749 30581 consensus_queue.cc:1035] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 330, Time since last communication: 0.000s
I20240503 02:13:20.542613 30581 raft_consensus.cc:2932] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 113 to 334, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:20.548105 29083 raft_consensus.cc:2932] T ddd409a1c4554665831eceef070ab660 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 113 to 334, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:20.569957 27511 catalog_manager.cc:5484] T ddd409a1c4554665831eceef070ab660 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 113 to 334, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:20.710892 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet ddd409a1c4554665831eceef070ab660 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 334) from {username='slave'} at 127.0.0.1:49186
I20240503 02:13:20.713840 30603 tablet_replica.cc:330] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:13:20.714429 30603 raft_consensus.cc:2229] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.714812 30603 pending_rounds.cc:62] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Trying to abort 1 pending ops.
I20240503 02:13:20.714941 30603 pending_rounds.cc:68] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Aborting op as it isn't in flight: id { term: 1 index: 333 } timestamp: 7023421032418783232 op_type: WRITE_OP write_request { tablet_id: "ddd409a1c4554665831eceef070ab660" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\035xY\'\004\260\333\215\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5b6e21b836cf8a33" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023421032035860480 authz_token { token_data: "\010\324\222\321\261\006\"3\n\005slave\022*\n 1397d007101b4cf6a7622c4a8aa5841c\020\001\030\001 \001(\001" signature: "<redacted>""+Q\323*J\355K\326\346\310\226\024\023nj\302|\013\363\013D\341v\265tO\344b\222Vd)\304J]Ik\275\346\272H\033\177\022\236\355\246\267\222\347\026\303\324\327\260\247\330\310\251\010\221\302\010\232" signing_key_seq_num: 0 } } request_id { client_id: "6fd88b80ccca41b2962b3d41716c3802" seq_no: 902 first_incomplete_seq_no: 902 attempt_no: 0 }
I20240503 02:13:20.715714 30603 raft_consensus.cc:2258] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.727931 30603 ts_tablet_manager.cc:1875] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:20.749531 30603 ts_tablet_manager.cc:1888] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.333
I20240503 02:13:20.749754 30603 log.cc:1196] T ddd409a1c4554665831eceef070ab660 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/ddd409a1c4554665831eceef070ab660
I20240503 02:13:20.753363 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet ddd409a1c4554665831eceef070ab660 (table rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]) successfully deleted
I20240503 02:13:21.589627 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:21.615655 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:21.656773 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:21.756246 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:21.773065 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:21.849673 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:21.856832 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:13:22.333585 27589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:46503, user_credentials={real_user=slave}} blocked reactor thread for 47840.9us
W20240503 02:13:22.520888 27592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 58213.6us
I20240503 02:13:22.609779 27654 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:22.615525 29343 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20240503 02:13:22.616653 30529 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20240503 02:13:22.623397 30452 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 95 to 320, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:22.626001 29341 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 95 to 320, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:22.631585 27511 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 95 to 320, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:22.749284 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 878e847516c74149966db83f6f70c6d0 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 320) from {username='slave'} at 127.0.0.1:51446
I20240503 02:13:22.786561 30685 tablet_replica.cc:330] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:13:22.786849 30685 raft_consensus.cc:2229] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.787147 30685 raft_consensus.cc:2258] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.790071 30685 ts_tablet_manager.cc:1875] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.851017 30685 ts_tablet_manager.cc:1888] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.319
I20240503 02:13:22.869419 30685 log.cc:1196] T 878e847516c74149966db83f6f70c6d0 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/878e847516c74149966db83f6f70c6d0
I20240503 02:13:22.879235 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet 878e847516c74149966db83f6f70c6d0 (table rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]) successfully deleted
I20240503 02:13:23.775489 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:23.818837 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.082549 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:24.111969 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:24.115921 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.145831 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:24.242141 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:25.103353 27654 consensus_queue.cc:237] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 379, Committed index: 379, Last appended: 1.380, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:25.109550 29083 raft_consensus.cc:1261] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch)
I20240503 02:13:25.118171 30610 consensus_queue.cc:1035] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 379, Time since last communication: 0.000s
I20240503 02:13:25.120659 30773 raft_consensus.cc:2932] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.381: config changed from index 141 to 381, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:25.125396 29083 raft_consensus.cc:2932] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 141 to 381, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:25.130393 27511 catalog_manager.cc:5484] T 6b842b202e3b4d17afd245f0d90a34a4 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 141 to 381, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:25.234799 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b842b202e3b4d17afd245f0d90a34a4 with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 381) from {username='slave'} at 127.0.0.1:40704
I20240503 02:13:25.235707 30786 tablet_replica.cc:330] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:13:25.236162 30786 raft_consensus.cc:2229] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:25.236574 30786 raft_consensus.cc:2258] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:25.237949 30786 ts_tablet_manager.cc:1875] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:25.250735 30786 ts_tablet_manager.cc:1888] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.379
I20240503 02:13:25.250949 30786 log.cc:1196] T 6b842b202e3b4d17afd245f0d90a34a4 P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:13:25.252187 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 6b842b202e3b4d17afd245f0d90a34a4 (table rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]) successfully deleted
I20240503 02:13:26.246513 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.316077 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.437415 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:26.471237 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.533130 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:13:26.553593 27592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 80877.8us
I20240503 02:13:26.567024 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:26.576812 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:27.464073 27784 consensus_queue.cc:237] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 457, Committed index: 457, Last appended: 1.457, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:27.467303 28949 raft_consensus.cc:1261] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 457. Preceding OpId from leader: term: 1 index: 458. (index mismatch)
I20240503 02:13:27.468577 30817 consensus_queue.cc:1035] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 458, Last known committed idx: 457, Time since last communication: 0.000s
I20240503 02:13:27.481510 30763 raft_consensus.cc:2932] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.458: config changed from index 123 to 458, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:27.484946 28950 raft_consensus.cc:2932] T 8bf2a0fe803b42e4a706f50762a4b87d P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Committing config change with OpId 1.458: config changed from index 123 to 458, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:27.530335 27510 catalog_manager.cc:5484] T 8bf2a0fe803b42e4a706f50762a4b87d P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 123 to 458, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 458 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:27.679497 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf2a0fe803b42e4a706f50762a4b87d with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 458) from {username='slave'} at 127.0.0.1:40704
I20240503 02:13:27.680967 30886 tablet_replica.cc:330] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:13:27.681253 30886 raft_consensus.cc:2229] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.681576 30886 raft_consensus.cc:2258] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.682713 30886 ts_tablet_manager.cc:1875] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.693014 30886 ts_tablet_manager.cc:1888] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.457
I20240503 02:13:27.693246 30886 log.cc:1196] T 8bf2a0fe803b42e4a706f50762a4b87d P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:13:27.699023 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 8bf2a0fe803b42e4a706f50762a4b87d (table rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]) successfully deleted
I20240503 02:13:28.687902 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.693703 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:28.742105 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:28.752146 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:28.753288 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.812984 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:28.831274 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:29.742502 27786 consensus_queue.cc:237] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 409, Committed index: 409, Last appended: 1.409, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:29.751864 29083 raft_consensus.cc:1261] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 409. Preceding OpId from leader: term: 1 index: 410. (index mismatch)
I20240503 02:13:29.754263 30817 consensus_queue.cc:1035] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 410, Last known committed idx: 409, Time since last communication: 0.000s
I20240503 02:13:29.766095 30763 raft_consensus.cc:2932] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.410: config changed from index 116 to 410, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:29.766782 29083 raft_consensus.cc:2932] T a52a6040dd89493da32af4b283774f03 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Committing config change with OpId 1.410: config changed from index 116 to 410, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:29.774778 27510 catalog_manager.cc:5484] T a52a6040dd89493da32af4b283774f03 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 116 to 410, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 410 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:29.996953 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet a52a6040dd89493da32af4b283774f03 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 410) from {username='slave'} at 127.0.0.1:49186
I20240503 02:13:30.016747 30973 tablet_replica.cc:330] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:13:30.017060 30973 raft_consensus.cc:2229] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:30.017364 30973 raft_consensus.cc:2258] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:30.020926 30973 ts_tablet_manager.cc:1875] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:30.034200 30973 ts_tablet_manager.cc:1888] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.409
I20240503 02:13:30.034960 30973 log.cc:1196] T a52a6040dd89493da32af4b283774f03 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/a52a6040dd89493da32af4b283774f03
I20240503 02:13:30.046406 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet a52a6040dd89493da32af4b283774f03 (table rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]) successfully deleted
I20240503 02:13:30.870985 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.883179 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:30.892668 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:30.960422 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.020205 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:31.122084 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:31.165706 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:31.843773 27917 consensus_queue.cc:237] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 534, Committed index: 534, Last appended: 1.534, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 535 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:31.853582 29344 raft_consensus.cc:1261] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 534. Preceding OpId from leader: term: 1 index: 535. (index mismatch)
I20240503 02:13:31.854359 30664 consensus_queue.cc:1035] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 535, Last known committed idx: 534, Time since last communication: 0.000s
I20240503 02:13:31.860826 30664 raft_consensus.cc:2932] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Committing config change with OpId 1.535: config changed from index 77 to 535, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 535 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:31.889966 29341 raft_consensus.cc:2932] T 1cc58692b4d144ed8bd4ad33cc8d022c P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.535: config changed from index 77 to 535, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 535 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:31.914836 27510 catalog_manager.cc:5484] T 1cc58692b4d144ed8bd4ad33cc8d022c P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index 77 to 535, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 535 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:31.968639 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cc58692b4d144ed8bd4ad33cc8d022c with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 535) from {username='slave'} at 127.0.0.1:49186
I20240503 02:13:31.972314 31065 tablet_replica.cc:330] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:13:31.972612 31065 raft_consensus.cc:2229] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.977052 31065 raft_consensus.cc:2258] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.980340 31065 ts_tablet_manager.cc:1875] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:32.004535 31065 ts_tablet_manager.cc:1888] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.534
I20240503 02:13:32.004770 31065 log.cc:1196] T 1cc58692b4d144ed8bd4ad33cc8d022c P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:13:32.008188 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet 1cc58692b4d144ed8bd4ad33cc8d022c (table rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]) successfully deleted
I20240503 02:13:32.805815 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.819089 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:32.880488 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:33.028654 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:33.088732 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:33.105003 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:33.187067 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:34.033555 27656 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 573, Committed index: 573, Last appended: 1.573, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 574 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:13:34.037562 29079 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 571. Preceding OpId from leader: term: 1 index: 574. (index mismatch)
I20240503 02:13:34.038905 30774 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 574, Last known committed idx: 571, Time since last communication: 0.000s
I20240503 02:13:34.043715 31028 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.574: config changed from index 81 to 574, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 574 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:34.045094 29079 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Committing config change with OpId 1.574: config changed from index 81 to 574, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 574 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:34.050323 27511 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 81 to 574, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 574 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } }
I20240503 02:13:34.108245 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 896ae927d34e46188120e705fb0ae361 with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 574) from {username='slave'} at 127.0.0.1:40704
I20240503 02:13:34.134469 29080 raft_consensus.cc:1203] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.571->[1.572-1.572]   Dedup: 1.572->[]
I20240503 02:13:34.160732 31157 tablet_replica.cc:330] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:13:34.160952 31157 raft_consensus.cc:2229] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:34.161180 31157 raft_consensus.cc:2258] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:34.162762 31157 ts_tablet_manager.cc:1875] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:34.186025 31157 ts_tablet_manager.cc:1888] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.573
I20240503 02:13:34.186777 31157 log.cc:1196] T 896ae927d34e46188120e705fb0ae361 P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/896ae927d34e46188120e705fb0ae361
I20240503 02:13:34.188136 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 896ae927d34e46188120e705fb0ae361 (table rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]) successfully deleted
I20240503 02:13:35.181301 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.314703 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.323117 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:35.357956 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:35.405532 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:35.492065 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:35.507082 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:36.218807 27914 consensus_queue.cc:237] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 1.516, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:36.228649 29212 raft_consensus.cc:1261] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20240503 02:13:36.234207 30869 consensus_queue.cc:1035] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20240503 02:13:36.264041 31235 raft_consensus.cc:2932] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 [term 1 LEADER]: Committing config change with OpId 1.517: config changed from index 48 to 517, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:36.281766 29212 raft_consensus.cc:2932] T f475da5fc141450ba4ddfe739ea3ad96 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 48 to 517, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:36.296067 27511 catalog_manager.cc:5484] T f475da5fc141450ba4ddfe739ea3ad96 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index 48 to 517, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:36.605316 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet f475da5fc141450ba4ddfe739ea3ad96 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 517) from {username='slave'} at 127.0.0.1:51446
I20240503 02:13:36.606129 31260 tablet_replica.cc:330] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:13:36.606736 31260 raft_consensus.cc:2229] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:36.606989 31260 raft_consensus.cc:2258] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:36.610241 31260 ts_tablet_manager.cc:1875] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:36.617622 31260 ts_tablet_manager.cc:1888] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.516
I20240503 02:13:36.624233 31260 log.cc:1196] T f475da5fc141450ba4ddfe739ea3ad96 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:13:36.626381 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet f475da5fc141450ba4ddfe739ea3ad96 (table rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]) successfully deleted
W20240503 02:13:37.467892 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:32881, user_credentials={real_user=slave}} blocked reactor thread for 43556.5us
I20240503 02:13:38.157011 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.574878 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.598814 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.714439 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:38.750810 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:38.756753 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:38.843608 27635 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:39.176174 27656 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a63be7e6f2da4163baced11894065444"
dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41218
I20240503 02:13:39.176524 27656 raft_consensus.cc:595] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:13:39.207743 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.217058 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.225615 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.236703 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.253563 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.265235 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.286489 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.314270 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.328905 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.355644 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.385567 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.404971 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.459590 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.478605 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.502300 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.523866 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.545238 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.567924 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.593835 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.624023 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.651703 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.692960 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
W20240503 02:13:39.725042 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Service unavailable: leader transfer in progress
I20240503 02:13:39.728693 31321 raft_consensus.cc:979] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: : Instructing follower acd475a9eee143a89c1d28a05fa51d26 to start an election
I20240503 02:13:39.728943 31321 raft_consensus.cc:1067] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Signalling peer acd475a9eee143a89c1d28a05fa51d26 to start an election
I20240503 02:13:39.730132 27787 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a63be7e6f2da4163baced11894065444"
dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
 from {username='slave'} at 127.20.218.65:38263
I20240503 02:13:39.730381 27787 raft_consensus.cc:482] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:39.730497 27787 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.734364 27787 raft_consensus.cc:504] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:39.736877 27656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 2 candidate_status { last_received { term: 1 index: 609 } } ignore_live_leader: true dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:13:39.737208 27656 raft_consensus.cc:3032] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:39.737365 27656 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:13:39.737762 27656 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 609, Committed index: 609, Last appended: 1.609, Last appended by leader: 609, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:39.738281 27656 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.742873 27656 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 2.
I20240503 02:13:39.743731 27722 leader_election.cc:302] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acd475a9eee143a89c1d28a05fa51d26, ecf926bd3fb84e0d8f66465d78a70d31; no voters: 
I20240503 02:13:39.744226 30817 raft_consensus.cc:2781] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:39.744457 30817 raft_consensus.cc:686] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Becoming Leader. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:13:39.744668 27787 leader_election.cc:288] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [CANDIDATE]: Term 2 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115), b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
I20240503 02:13:39.744783 30817 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 609, Committed index: 609, Last appended: 1.609, Last appended by leader: 609, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:39.745759 28952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "acd475a9eee143a89c1d28a05fa51d26" candidate_term: 2 candidate_status { last_received { term: 1 index: 609 } } ignore_live_leader: true dest_uuid: "b0eb49c52e794992a8fc34173f0cdcef"
I20240503 02:13:39.746059 28952 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:39.749987 28952 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd475a9eee143a89c1d28a05fa51d26 in term 2.
I20240503 02:13:39.749570 27510 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: term changed from 1 to 2, leader changed from ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) to acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66). New cstate: current_term: 2 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
W20240503 02:13:39.759331 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42510: Illegal state: replica ecf926bd3fb84e0d8f66465d78a70d31 is not leader of this config: current role FOLLOWER
I20240503 02:13:39.792126 28952 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 609. Preceding OpId from leader: term: 2 index: 611. (index mismatch)
I20240503 02:13:39.792143 27656 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 609. Preceding OpId from leader: term: 2 index: 611. (index mismatch)
I20240503 02:13:39.792987 30817 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 610, Last known committed idx: 609, Time since last communication: 0.000s
I20240503 02:13:39.798478 31313 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 610, Last known committed idx: 609, Time since last communication: 0.000s
I20240503 02:13:39.799939 30942 mvcc.cc:205] Tried to move back new op lower bound from 7023421111460786176 to 7023421111277731840. Current Snapshot: MvccSnapshot[applied={T|T < 7023421111460786176 or (T in {7023421111460786176})}]
I20240503 02:13:39.944219 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:39.995540 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:40.011209 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:40.068015 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:13:40.179658 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:32881, user_credentials={real_user=slave}} blocked reactor thread for 57646us
I20240503 02:13:40.192216 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:40.202081 27635 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:40.239821 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:41.344156 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:41.354810 27635 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:41.394615 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:41.409461 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:41.429649 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:41.434883 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:41.487011 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:13:41.920312 27589 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:46503, user_credentials={real_user=slave}} blocked reactor thread for 102252us
I20240503 02:13:42.490448 27656 consensus_queue.cc:237] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 655, Committed index: 655, Last appended: 1.655, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:42.516391 29341 raft_consensus.cc:1261] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 655. Preceding OpId from leader: term: 1 index: 656. (index mismatch)
I20240503 02:13:42.520983 31320 consensus_queue.cc:1035] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 656, Last known committed idx: 655, Time since last communication: 0.000s
I20240503 02:13:42.526680 31452 raft_consensus.cc:2932] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.656: config changed from index 81 to 656, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:42.536801 27510 catalog_manager.cc:5484] T 376511df6d8645268134e125791f6aca P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 81 to 656, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:42.537961 29341 raft_consensus.cc:2932] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.656: config changed from index 81 to 656, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:42.693854 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 376511df6d8645268134e125791f6aca with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 656) from {username='slave'} at 127.0.0.1:51446
I20240503 02:13:42.694572 31501 tablet_replica.cc:330] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:13:42.694801 31501 raft_consensus.cc:2229] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:42.694998 31501 raft_consensus.cc:2258] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:42.696065 31501 ts_tablet_manager.cc:1875] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:42.707111 31501 ts_tablet_manager.cc:1888] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.655
I20240503 02:13:42.707383 31501 log.cc:1196] T 376511df6d8645268134e125791f6aca P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/376511df6d8645268134e125791f6aca
I20240503 02:13:42.708815 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet 376511df6d8645268134e125791f6aca (table rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]) successfully deleted
I20240503 02:13:43.467257 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:43.478082 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:43.634464 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:43.664417 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:43.773708 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:43.878528 27635 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:43.912817 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:44.455471 27787 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 617, Committed index: 617, Last appended: 1.617, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:44.458966 29212 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 617. Preceding OpId from leader: term: 1 index: 618. (index mismatch)
I20240503 02:13:44.465734 31503 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 618, Last known committed idx: 617, Time since last communication: 0.000s
I20240503 02:13:44.493567 31313 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.618: config changed from index 88 to 618, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:44.499428 29212 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.618: config changed from index 88 to 618, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:44.548676 27514 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 reported cstate change: config changed from index 88 to 618, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:44.586197 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec65a7b8d0943119b929ae3a9544ec0 with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 618) from {username='slave'} at 127.0.0.1:40704
I20240503 02:13:44.588681 31575 tablet_replica.cc:330] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:13:44.588948 31575 raft_consensus.cc:2229] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:44.589171 31575 raft_consensus.cc:2258] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:44.591125 31575 ts_tablet_manager.cc:1875] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:44.603503 31575 ts_tablet_manager.cc:1888] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.617
I20240503 02:13:44.603749 31575 log.cc:1196] T 4ec65a7b8d0943119b929ae3a9544ec0 P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:13:44.606185 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 4ec65a7b8d0943119b929ae3a9544ec0 (table rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]) successfully deleted
I20240503 02:13:45.942765 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.955745 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:45.968672 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:46.039862 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:46.057121 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:13:46.106580 27635 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:46.144340 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:46.440112 27654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4395a743162549ca96757c5838c0e190"
dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41180
I20240503 02:13:46.440385 27654 raft_consensus.cc:595] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:13:46.493083 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:32881, user_credentials={real_user=slave}} blocked reactor thread for 45564.3us
W20240503 02:13:46.590179 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.598038 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.610448 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.625936 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.633035 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.646173 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.658321 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.673745 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.685114 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.698590 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.717123 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.734619 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.749807 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.768244 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.786382 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.809854 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.834925 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
W20240503 02:13:46.859658 27613 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
I20240503 02:13:46.884349 31454 raft_consensus.cc:979] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: : Instructing follower eaffa047cf9b43b79a5b75089ea63901 to start an election
I20240503 02:13:46.884563 31454 raft_consensus.cc:1067] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Signalling peer eaffa047cf9b43b79a5b75089ea63901 to start an election
W20240503 02:13:46.886230 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Service unavailable: leader transfer in progress
I20240503 02:13:46.894421 27915 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4395a743162549ca96757c5838c0e190"
dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
 from {username='slave'} at 127.20.218.65:44729
I20240503 02:13:46.894903 27915 raft_consensus.cc:482] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:46.895272 27915 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:46.899410 27915 raft_consensus.cc:504] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:46.902943 27915 leader_election.cc:288] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 2 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115), b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
I20240503 02:13:46.903641 28951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 2 candidate_status { last_received { term: 1 index: 647 } } ignore_live_leader: true dest_uuid: "b0eb49c52e794992a8fc34173f0cdcef"
I20240503 02:13:46.903661 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 2 candidate_status { last_received { term: 1 index: 647 } } ignore_live_leader: true dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:13:46.903975 28951 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:46.903990 27654 raft_consensus.cc:3032] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:46.904094 27654 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:13:46.904435 27654 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 647, Committed index: 647, Last appended: 1.647, Last appended by leader: 647, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:46.904897 27654 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:46.908370 28951 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 2.
I20240503 02:13:46.909917 27852 leader_election.cc:302] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0eb49c52e794992a8fc34173f0cdcef, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:13:46.911072 31653 raft_consensus.cc:2781] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:46.911270 31653 raft_consensus.cc:686] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Becoming Leader. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
W20240503 02:13:46.911679 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42502: Illegal state: replica ecf926bd3fb84e0d8f66465d78a70d31 is not leader of this config: current role FOLLOWER
I20240503 02:13:46.911626 31653 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 647, Committed index: 647, Last appended: 1.647, Last appended by leader: 647, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:46.915568 27514 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: term changed from 1 to 2, leader changed from ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) to eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:46.931475 27654 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 2.
I20240503 02:13:47.007928 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:47.009635 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:47.070533 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:47.109640 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:47.151494 27654 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 647. Preceding OpId from leader: term: 2 index: 649. (index mismatch)
I20240503 02:13:47.151760 28951 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 647. Preceding OpId from leader: term: 2 index: 649. (index mismatch)
I20240503 02:13:47.152367 31667 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 648, Last known committed idx: 647, Time since last communication: 0.000s
I20240503 02:13:47.152774 31653 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 648, Last known committed idx: 647, Time since last communication: 0.000s
I20240503 02:13:47.225498 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:47.227061 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:47.584213 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:49.716838 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:49.722807 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:49.900421 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:50.022969 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:50.090446 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:50.093762 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:50.098837 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:50.899708 27787 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 782, Committed index: 782, Last appended: 2.782, Last appended by leader: 609, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 783 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:50.902776 28952 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 2 index: 782. Preceding OpId from leader: term: 2 index: 783. (index mismatch)
I20240503 02:13:50.903703 31503 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 783, Last known committed idx: 782, Time since last communication: 0.000s
I20240503 02:13:50.909050 31503 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Committing config change with OpId 2.783: config changed from index 90 to 783, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 783 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:50.911296 28952 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Committing config change with OpId 2.783: config changed from index 90 to 783, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 783 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:50.951579 27511 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 90 to 783, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 783 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:51.112951 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet a63be7e6f2da4163baced11894065444 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 783) from {username='slave'} at 127.0.0.1:49186
I20240503 02:13:51.113783 31880 tablet_replica.cc:330] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:13:51.114053 31880 raft_consensus.cc:2229] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:51.114526 31880 raft_consensus.cc:2258] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:51.115669 31880 ts_tablet_manager.cc:1875] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:51.129590 31880 ts_tablet_manager.cc:1888] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.782
I20240503 02:13:51.129871 31880 log.cc:1196] T a63be7e6f2da4163baced11894065444 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/a63be7e6f2da4163baced11894065444
I20240503 02:13:51.131630 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet a63be7e6f2da4163baced11894065444 (table rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]) successfully deleted
I20240503 02:13:52.308826 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.327389 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.425104 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.525390 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:52.525532 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:52.626399 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:52.667382 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:53.445792 27913 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 769, Committed index: 769, Last appended: 2.769, Last appended by leader: 647, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 770 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } }
I20240503 02:13:53.448001 28950 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 2 index: 769. Preceding OpId from leader: term: 2 index: 770. (index mismatch)
I20240503 02:13:53.449121 31712 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 770, Last known committed idx: 769, Time since last communication: 0.000s
I20240503 02:13:53.452086 31712 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Committing config change with OpId 2.770: config changed from index 126 to 770, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 770 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:53.459231 28950 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Committing config change with OpId 2.770: config changed from index 126 to 770, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 770 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
I20240503 02:13:53.465006 27511 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index 126 to 770, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 770 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } }
W20240503 02:13:53.536355 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 47263.7us
W20240503 02:13:53.611938 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:32881, user_credentials={real_user=slave}} blocked reactor thread for 57262.9us
I20240503 02:13:53.690966 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet 4395a743162549ca96757c5838c0e190 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 770) from {username='slave'} at 127.0.0.1:49186
I20240503 02:13:53.815451 31987 tablet_replica.cc:330] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:13:53.815810 31987 raft_consensus.cc:2229] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:53.818761 31987 raft_consensus.cc:2258] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:53.820348 31987 ts_tablet_manager.cc:1875] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:53.829471 31987 ts_tablet_manager.cc:1888] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.769
I20240503 02:13:53.830947 31987 log.cc:1196] T 4395a743162549ca96757c5838c0e190 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/4395a743162549ca96757c5838c0e190
I20240503 02:13:53.832366 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet 4395a743162549ca96757c5838c0e190 (table rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]) successfully deleted
I20240503 02:13:54.972239 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:54.992069 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:55.015233 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:55.030332 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:55.102511 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:55.189944 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:55.190855 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:55.743041 27787 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 859, Committed index: 859, Last appended: 2.859, Last appended by leader: 609, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:55.746165 28952 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 2 index: 859. Preceding OpId from leader: term: 2 index: 860. (index mismatch)
I20240503 02:13:55.746965 31503 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 860, Last known committed idx: 859, Time since last communication: 0.000s
W20240503 02:13:55.752111 27723 consensus_peers.cc:485] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a63be7e6f2da4163baced11894065444. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:55.756966 28950 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Committing config change with OpId 2.860: config changed from index 783 to 860, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:55.752949 31386 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Committing config change with OpId 2.860: config changed from index 783 to 860, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:55.770409 27514 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index 783 to 860, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New cstate: current_term: 2 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:56.221578 27654 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 767, Committed index: 767, Last appended: 1.767, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:56.224562 29341 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 767. Preceding OpId from leader: term: 1 index: 768. (index mismatch)
I20240503 02:13:56.226011 31959 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 768, Last known committed idx: 767, Time since last communication: 0.000s
I20240503 02:13:56.232998 31959 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.768: config changed from index 320 to 768, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:56.239485 29341 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.768: config changed from index 320 to 768, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New config: { opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:13:56.253496 27509 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 320 to 768, VOTER 24868c13491843548e8da6583ef1e9d5 (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
W20240503 02:13:56.272194 27592 consensus_peers.cc:485] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 878e847516c74149966db83f6f70c6d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:56.433976 32122 ts_tablet_manager.cc:902] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:13:56.516842 32122 tablet_copy_client.cc:308] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:13:56.519519 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet a63be7e6f2da4163baced11894065444 from peer 24868c13491843548e8da6583ef1e9d5 ({username='slave'} at 127.20.218.70:38931)
I20240503 02:13:56.519737 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet a63be7e6f2da4163baced11894065444 from peer 24868c13491843548e8da6583ef1e9d5 at {username='slave'} at 127.20.218.70:38931: session id = 24868c13491843548e8da6583ef1e9d5-a63be7e6f2da4163baced11894065444
I20240503 02:13:56.522632 27807 tablet_copy_source_session.cc:205] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:56.531970 32122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63be7e6f2da4163baced11894065444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:56.541226 32122 tablet_copy_client.cc:783] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:56.541700 32122 tablet_copy_client.cc:647] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:56.554296 32122 tablet_copy_client.cc:523] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:56.576521 32122 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Bootstrap starting.
I20240503 02:13:56.717383 27787 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 777, Committed index: 777, Last appended: 1.777, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:13:56.721925 29213 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 777. Preceding OpId from leader: term: 1 index: 778. (index mismatch)
I20240503 02:13:56.722832 31503 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 778, Last known committed idx: 777, Time since last communication: 0.000s
I20240503 02:13:56.726624 31386 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.778: config changed from index 618 to 778, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:56.732362 29213 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.778: config changed from index 618 to 778, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New config: { opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
W20240503 02:13:56.754539 27720 consensus_peers.cc:485] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ec65a7b8d0943119b929ae3a9544ec0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:56.761572 27509 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 reported cstate change: config changed from index 618 to 778, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:13:56.843652 32143 ts_tablet_manager.cc:902] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:13:57.166301 32143 tablet_copy_client.cc:308] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:13:57.224573 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 878e847516c74149966db83f6f70c6d0 from peer 24868c13491843548e8da6583ef1e9d5 ({username='slave'} at 127.20.218.70:33371)
I20240503 02:13:57.224854 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 878e847516c74149966db83f6f70c6d0 from peer 24868c13491843548e8da6583ef1e9d5 at {username='slave'} at 127.20.218.70:33371: session id = 24868c13491843548e8da6583ef1e9d5-878e847516c74149966db83f6f70c6d0
I20240503 02:13:57.228984 27676 tablet_copy_source_session.cc:205] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:57.232527 32143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 878e847516c74149966db83f6f70c6d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:57.245721 32143 tablet_copy_client.cc:783] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:57.246289 32143 tablet_copy_client.cc:647] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:57.253710 32143 tablet_copy_client.cc:523] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:57.269580 32143 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap starting.
I20240503 02:13:57.353555 32162 ts_tablet_manager.cc:902] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:13:57.611970 32162 tablet_copy_client.cc:308] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
W20240503 02:13:57.616856 32122 log.cc:924] Time spent T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
I20240503 02:13:57.883385 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet 4ec65a7b8d0943119b929ae3a9544ec0 from peer 94278d0a47dc47f2a0d5be9589f641be ({username='slave'} at 127.20.218.71:49161)
I20240503 02:13:57.883586 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet 4ec65a7b8d0943119b929ae3a9544ec0 from peer 94278d0a47dc47f2a0d5be9589f641be at {username='slave'} at 127.20.218.71:49161: session id = 94278d0a47dc47f2a0d5be9589f641be-4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:13:57.886497 27807 tablet_copy_source_session.cc:205] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:57.890872 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ec65a7b8d0943119b929ae3a9544ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:57.899752 32162 tablet_copy_client.cc:783] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:57.901510 32162 tablet_copy_client.cc:647] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:57.930050 32162 tablet_copy_client.cc:523] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:57.939702 32162 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap starting.
W20240503 02:13:58.581840 27723 consensus_peers.cc:485] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:58.654034 27592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 64480.8us
I20240503 02:13:58.738636 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.781512 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.919915 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:58.924047 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:13:59.042333 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:13:59.170782 32122 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=870 overwritten=0 applied=870 ignored=0} inserts{seen=1539 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:59.171519 32122 tablet_bootstrap.cc:492] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Bootstrap complete.
I20240503 02:13:59.172008 32122 ts_tablet_manager.cc:1367] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Time spent bootstrapping tablet: real 2.596s	user 1.093s	sys 0.143s
I20240503 02:13:59.172991 32122 raft_consensus.cc:348] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:59.173720 32122 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24868c13491843548e8da6583ef1e9d5, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.174075 32122 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 870, Last appended: 2.870, Last appended by leader: 870, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
W20240503 02:13:59.184386 27592 consensus_peers.cc:485] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393): Couldn't send request to peer 24868c13491843548e8da6583ef1e9d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:59.186076 32122 ts_tablet_manager.cc:1398] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.008s
I20240503 02:13:59.186975 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session 24868c13491843548e8da6583ef1e9d5-a63be7e6f2da4163baced11894065444 received from {username='slave'} at 127.20.218.70:38931
I20240503 02:13:59.187144 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session 24868c13491843548e8da6583ef1e9d5-a63be7e6f2da4163baced11894065444 on tablet a63be7e6f2da4163baced11894065444 with peer 24868c13491843548e8da6583ef1e9d5
I20240503 02:13:59.188719 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:13:59.277716 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:13:59.604535 29211 raft_consensus.cc:1203] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.859->[2.860-2.917]   Dedup: 2.870->[2.871-2.917]
I20240503 02:13:59.829178 32143 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=784 overwritten=0 applied=783 ignored=0} inserts{seen=1466 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:13:59.830010 32143 tablet_bootstrap.cc:492] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Bootstrap complete.
I20240503 02:13:59.830438 32143 ts_tablet_manager.cc:1367] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Time spent bootstrapping tablet: real 2.561s	user 0.964s	sys 0.082s
I20240503 02:13:59.831259 32143 raft_consensus.cc:348] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:59.832417 32143 raft_consensus.cc:729] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24868c13491843548e8da6583ef1e9d5, State: Initialized, Role: FOLLOWER
I20240503 02:13:59.832773 32143 consensus_queue.cc:260] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 783, Last appended: 1.784, Last appended by leader: 784, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:13:59.833527 32143 ts_tablet_manager.cc:1398] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:13:59.841620 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 24868c13491843548e8da6583ef1e9d5-878e847516c74149966db83f6f70c6d0 received from {username='slave'} at 127.20.218.70:33371
I20240503 02:13:59.841773 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 24868c13491843548e8da6583ef1e9d5-878e847516c74149966db83f6f70c6d0 on tablet 878e847516c74149966db83f6f70c6d0 with peer 24868c13491843548e8da6583ef1e9d5
W20240503 02:14:00.017446 27720 consensus_peers.cc:485] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): Couldn't send request to peer 94278d0a47dc47f2a0d5be9589f641be. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:00.103924 29210 raft_consensus.cc:1203] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.767->[1.768-1.811]   Dedup: 1.784->[1.785-1.811]
W20240503 02:14:00.166347 27723 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:45743, user_credentials={real_user=slave}} blocked reactor thread for 70544.7us
I20240503 02:14:00.351954 32162 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap replayed 1/1 log segments. Stats: ops{read=790 overwritten=0 applied=790 ignored=0} inserts{seen=1419 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:00.352891 32162 tablet_bootstrap.cc:492] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Bootstrap complete.
I20240503 02:14:00.353508 32162 ts_tablet_manager.cc:1367] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Time spent bootstrapping tablet: real 2.414s	user 0.958s	sys 0.089s
I20240503 02:14:00.354579 32162 raft_consensus.cc:348] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:00.354876 32162 raft_consensus.cc:729] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Initialized, Role: FOLLOWER
I20240503 02:14:00.355172 32162 consensus_queue.cc:260] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 790, Last appended: 1.790, Last appended by leader: 790, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:00.364198 32162 ts_tablet_manager.cc:1398] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20240503 02:14:00.366484 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session 94278d0a47dc47f2a0d5be9589f641be-4ec65a7b8d0943119b929ae3a9544ec0 received from {username='slave'} at 127.20.218.71:49161
I20240503 02:14:00.366669 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session 94278d0a47dc47f2a0d5be9589f641be-4ec65a7b8d0943119b929ae3a9544ec0 on tablet 4ec65a7b8d0943119b929ae3a9544ec0 with peer 94278d0a47dc47f2a0d5be9589f641be
W20240503 02:14:00.392505 27722 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:39319, user_credentials={real_user=slave}} blocked reactor thread for 88972.8us
I20240503 02:14:00.611757 29344 raft_consensus.cc:1203] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.777->[1.778-1.820]   Dedup: 1.790->[1.791-1.820]
I20240503 02:14:00.738911 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:00.765301 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:00.809296 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:00.855343 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:00.938462 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:00.982919 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:01.073742 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:02.147560 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:02.198668 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:02.234208 27635 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:02.235865 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:02.260291 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:02.274869 29324 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:02.414464 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:03.003075 27654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "878e847516c74149966db83f6f70c6d0"
dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50548
I20240503 02:14:03.003312 27654 raft_consensus.cc:595] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:14:03.007783 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
W20240503 02:14:03.012581 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
W20240503 02:14:03.020995 27614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
W20240503 02:14:03.030758 27614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
W20240503 02:14:03.042361 27614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
I20240503 02:14:03.049727 32273 raft_consensus.cc:979] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: : Instructing follower 94278d0a47dc47f2a0d5be9589f641be to start an election
I20240503 02:14:03.049913 32273 raft_consensus.cc:1067] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Signalling peer 94278d0a47dc47f2a0d5be9589f641be to start an election
I20240503 02:14:03.053496 29344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "878e847516c74149966db83f6f70c6d0"
dest_uuid: "94278d0a47dc47f2a0d5be9589f641be"
 from {username='slave'} at 127.20.218.65:43073
I20240503 02:14:03.053834 29344 raft_consensus.cc:482] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:03.055258 29344 raft_consensus.cc:3037] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:03.056452 27614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Service unavailable: leader transfer in progress
I20240503 02:14:03.061677 29344 raft_consensus.cc:504] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:03.065601 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878e847516c74149966db83f6f70c6d0" candidate_uuid: "94278d0a47dc47f2a0d5be9589f641be" candidate_term: 2 candidate_status { last_received { term: 1 index: 846 } } ignore_live_leader: true dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:14:03.065865 27654 raft_consensus.cc:3032] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:03.065964 27654 raft_consensus.cc:729] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:14:03.066246 27654 consensus_queue.cc:260] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 846, Committed index: 846, Last appended: 1.846, Last appended by leader: 846, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:03.066624 27654 raft_consensus.cc:3037] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:14:03.073218 27615 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Illegal state: replica ecf926bd3fb84e0d8f66465d78a70d31 is not leader of this config: current role FOLLOWER
I20240503 02:14:03.085816 27654 raft_consensus.cc:2454] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94278d0a47dc47f2a0d5be9589f641be in term 2.
I20240503 02:14:03.093761 29344 leader_election.cc:288] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115), 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393)
I20240503 02:14:03.124738 29213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "878e847516c74149966db83f6f70c6d0" candidate_uuid: "94278d0a47dc47f2a0d5be9589f641be" candidate_term: 2 candidate_status { last_received { term: 1 index: 846 } } ignore_live_leader: true dest_uuid: "24868c13491843548e8da6583ef1e9d5"
I20240503 02:14:03.125200 29213 raft_consensus.cc:3037] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:03.128676 29213 raft_consensus.cc:2454] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94278d0a47dc47f2a0d5be9589f641be in term 2.
I20240503 02:14:03.131469 29280 leader_election.cc:302] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24868c13491843548e8da6583ef1e9d5, 94278d0a47dc47f2a0d5be9589f641be; no voters: 
I20240503 02:14:03.145108 32402 raft_consensus.cc:2781] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:03.153952 32402 raft_consensus.cc:686] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 LEADER]: Becoming Leader. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Running, Role: LEADER
I20240503 02:14:03.155158 32402 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 846, Committed index: 846, Last appended: 1.846, Last appended by leader: 846, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:03.160226 27509 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be reported cstate change: term changed from 1 to 2, leader changed from ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) to 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71). New cstate: current_term: 2 leader_uuid: "94278d0a47dc47f2a0d5be9589f641be" committed_config { opid_index: 768 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
W20240503 02:14:03.199276 27616 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42492: Illegal state: replica ecf926bd3fb84e0d8f66465d78a70d31 is not leader of this config: current role FOLLOWER
I20240503 02:14:03.313263 29211 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 1 index: 846. Preceding OpId from leader: term: 2 index: 848. (index mismatch)
I20240503 02:14:03.313843 27654 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 1 index: 846. Preceding OpId from leader: term: 2 index: 848. (index mismatch)
I20240503 02:14:03.314489 32403 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 847, Last known committed idx: 846, Time since last communication: 0.000s
I20240503 02:14:03.326134 32228 mvcc.cc:205] Tried to move back new op lower bound from 7023421207802535936 to 7023421207169015808. Current Snapshot: MvccSnapshot[applied={T|T < 7023421207802535936 or (T in {7023421207802535936})}]
I20240503 02:14:03.332369 32403 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 847, Last known committed idx: 846, Time since last communication: 0.000s
W20240503 02:14:03.678252 29280 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:37393, user_credentials={real_user=slave}} blocked reactor thread for 81636us
I20240503 02:14:04.163848 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:04.189921 29324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:04.207739 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:04.280654 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:04.321199 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:04.363144 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:04.582073 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:14:04.969307 27839 kernel_stack_watchdog.cc:198] Thread 27852 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 112ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f87dc8f8980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7f87d9566d11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7f87d95877ab  kudu::ThreadPool::CreateThread() at ??:0
    @     0x7f87d95833ad  kudu::ThreadPool::DoSubmit() at ??:0
    @     0x7f87d958184f  kudu::ThreadPoolToken::Submit() at ??:0
    @     0x7f87e4cb49e8  kudu::consensus::Peer::ProcessResponse() at ??:0
    @     0x7f87e4cbb061  kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
    @     0x7f87e4cbae6e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f87e01b88cf  kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
    @     0x7f87e01b84ae  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f87e01a44e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
W20240503 02:14:04.987133 27852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:39319, user_credentials={real_user=slave}} blocked reactor thread for 252963us
I20240503 02:14:05.540302 29341 consensus_queue.cc:237] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 882, Committed index: 882, Last appended: 2.882, Last appended by leader: 846, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 883 OBSOLETE_local: false peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:05.542833 29212 raft_consensus.cc:1261] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 2 index: 882. Preceding OpId from leader: term: 2 index: 883. (index mismatch)
I20240503 02:14:05.544579 32430 consensus_queue.cc:1035] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 883, Last known committed idx: 882, Time since last communication: 0.000s
I20240503 02:14:05.550539 32430 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 LEADER]: Committing config change with OpId 2.883: config changed from index 768 to 883, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 883 OBSOLETE_local: false peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:05.555697 29212 raft_consensus.cc:2932] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Committing config change with OpId 2.883: config changed from index 768 to 883, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 883 OBSOLETE_local: false peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:05.592990 27509 catalog_manager.cc:5484] T 878e847516c74149966db83f6f70c6d0 P 24868c13491843548e8da6583ef1e9d5 reported cstate change: config changed from index 768 to 883, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "94278d0a47dc47f2a0d5be9589f641be" committed_config { opid_index: 883 OBSOLETE_local: false peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:05.729022 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet 878e847516c74149966db83f6f70c6d0 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 883) from {username='slave'} at 127.0.0.1:49186
I20240503 02:14:05.768918 32537 tablet_replica.cc:330] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:14:05.769145 32537 raft_consensus.cc:2229] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:05.769439 32537 raft_consensus.cc:2258] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:05.776767 32537 ts_tablet_manager.cc:1875] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:05.919296 32537 ts_tablet_manager.cc:1888] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.882
I20240503 02:14:05.919435 32537 log.cc:1196] T 878e847516c74149966db83f6f70c6d0 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/878e847516c74149966db83f6f70c6d0
I20240503 02:14:05.928906 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet 878e847516c74149966db83f6f70c6d0 (table rebalance_test_table_0 [id=61134bde9f9b4a2c911dfdd0db516378]) successfully deleted
I20240503 02:14:06.549487 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:06.641223 27766 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:14:06.770053 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:06.822233 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:06.830399 29324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:06.912441 28932 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:06.983517 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:07.169162 27787 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a63be7e6f2da4163baced11894065444"
dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55888
I20240503 02:14:07.169466 27787 raft_consensus.cc:595] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Received request to transfer leadership
W20240503 02:14:07.174666 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.184886 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.197471 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.221194 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.288149 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.305313 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.314729 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.326207 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.336418 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.351508 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.365731 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.381551 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.399938 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.419536 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.437849 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.454927 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.473973 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.497828 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.517812 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.543294 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.566936 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.593725 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.625416 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
W20240503 02:14:07.654158 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
I20240503 02:14:07.669720 32383 raft_consensus.cc:979] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: : Instructing follower b0eb49c52e794992a8fc34173f0cdcef to start an election
I20240503 02:14:07.670060 32383 raft_consensus.cc:1067] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Signalling peer b0eb49c52e794992a8fc34173f0cdcef to start an election
I20240503 02:14:07.671754 28951 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a63be7e6f2da4163baced11894065444"
dest_uuid: "b0eb49c52e794992a8fc34173f0cdcef"
 from {username='slave'} at 127.20.218.66:33691
I20240503 02:14:07.672104 28951 raft_consensus.cc:482] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:07.672235 28951 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:07.675181 28951 raft_consensus.cc:504] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:07.684649 27787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "b0eb49c52e794992a8fc34173f0cdcef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1019 } } ignore_live_leader: true dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
W20240503 02:14:07.684870 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Service unavailable: leader transfer in progress
I20240503 02:14:07.685007 27787 raft_consensus.cc:3032] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:14:07.685134 27787 raft_consensus.cc:729] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:14:07.685573 27787 consensus_queue.cc:260] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1019, Committed index: 1019, Last appended: 2.1019, Last appended by leader: 1019, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:07.686081 27787 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:07.692595 28951 leader_election.cc:288] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [CANDIDATE]: Term 3 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503), 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393)
I20240503 02:14:07.698153 27787 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0eb49c52e794992a8fc34173f0cdcef in term 3.
I20240503 02:14:07.708470 29210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63be7e6f2da4163baced11894065444" candidate_uuid: "b0eb49c52e794992a8fc34173f0cdcef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1019 } } ignore_live_leader: true dest_uuid: "24868c13491843548e8da6583ef1e9d5"
I20240503 02:14:07.708827 29210 raft_consensus.cc:3037] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:07.711964 29210 raft_consensus.cc:2454] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0eb49c52e794992a8fc34173f0cdcef in term 3.
I20240503 02:14:07.712608 28888 leader_election.cc:302] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24868c13491843548e8da6583ef1e9d5, b0eb49c52e794992a8fc34173f0cdcef; no voters: 
W20240503 02:14:07.730062 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46292: Illegal state: replica acd475a9eee143a89c1d28a05fa51d26 is not leader of this config: current role FOLLOWER
I20240503 02:14:07.740598 32639 raft_consensus.cc:2781] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:14:07.756956 32639 raft_consensus.cc:686] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 3 LEADER]: Becoming Leader. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Running, Role: LEADER
I20240503 02:14:07.757687 32639 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1019, Committed index: 1019, Last appended: 2.1019, Last appended by leader: 1019, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:07.769901 27509 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef reported cstate change: term changed from 2 to 3, leader changed from acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) to b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68). New cstate: current_term: 3 leader_uuid: "b0eb49c52e794992a8fc34173f0cdcef" committed_config { opid_index: 860 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:07.915403 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:07.972990 29324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:08.071455 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:08.137650 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:08.159786 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:08.178364 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:08.185184 29210 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1019. Preceding OpId from leader: term: 3 index: 1020. (index mismatch)
I20240503 02:14:08.186437 32640 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1020, Last known committed idx: 1019, Time since last communication: 0.000s
I20240503 02:14:08.214365 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:08.249542 27787 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1019. Preceding OpId from leader: term: 3 index: 1020. (index mismatch)
I20240503 02:14:08.268604 32640 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1020, Last known committed idx: 1019, Time since last communication: 0.000s
I20240503 02:14:09.770506 27766 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:14:09.787622 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:09.813625 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:09.841509 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:09.890987 29324 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:09.925475 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:10.020529 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:10.557978 27786 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0"
dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:50874
I20240503 02:14:10.558233 27786 raft_consensus.cc:595] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Received request to transfer leadership
W20240503 02:14:10.587920 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.591778 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.598718 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.608744 27746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.624475 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.644918 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.665951 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.686347 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.711547 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.746301 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.768647 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
W20240503 02:14:10.788872 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Service unavailable: leader transfer in progress
I20240503 02:14:10.800420 32383 raft_consensus.cc:979] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: : Instructing follower 94278d0a47dc47f2a0d5be9589f641be to start an election
I20240503 02:14:10.800632 32383 raft_consensus.cc:1067] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Signalling peer 94278d0a47dc47f2a0d5be9589f641be to start an election
I20240503 02:14:10.802225 29341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0"
dest_uuid: "94278d0a47dc47f2a0d5be9589f641be"
 from {username='slave'} at 127.20.218.66:59031
I20240503 02:14:10.802537 29341 raft_consensus.cc:482] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:10.802664 29341 raft_consensus.cc:3037] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:10.805917 29341 raft_consensus.cc:504] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:10.807384 29341 leader_election.cc:288] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503), 24868c13491843548e8da6583ef1e9d5 (127.20.218.70:37393)
I20240503 02:14:10.807909 29212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0" candidate_uuid: "94278d0a47dc47f2a0d5be9589f641be" candidate_term: 2 candidate_status { last_received { term: 1 index: 960 } } ignore_live_leader: true dest_uuid: "24868c13491843548e8da6583ef1e9d5"
I20240503 02:14:10.807925 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ec65a7b8d0943119b929ae3a9544ec0" candidate_uuid: "94278d0a47dc47f2a0d5be9589f641be" candidate_term: 2 candidate_status { last_received { term: 1 index: 960 } } ignore_live_leader: true dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:14:10.808189 29212 raft_consensus.cc:3037] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:10.808207 27786 raft_consensus.cc:3032] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:10.808321 27786 raft_consensus.cc:729] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
W20240503 02:14:10.811134 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Illegal state: replica acd475a9eee143a89c1d28a05fa51d26 is not leader of this config: current role FOLLOWER
I20240503 02:14:10.811744 29212 raft_consensus.cc:2454] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94278d0a47dc47f2a0d5be9589f641be in term 2.
I20240503 02:14:10.812768 29280 leader_election.cc:302] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24868c13491843548e8da6583ef1e9d5, 94278d0a47dc47f2a0d5be9589f641be; no voters: 
I20240503 02:14:10.813124 32706 raft_consensus.cc:2781] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:10.813925 27786 consensus_queue.cc:260] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 960, Committed index: 960, Last appended: 1.960, Last appended by leader: 960, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:10.819620 27786 raft_consensus.cc:3037] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:10.828886 27786 raft_consensus.cc:2454] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94278d0a47dc47f2a0d5be9589f641be in term 2.
I20240503 02:14:10.828943 32706 raft_consensus.cc:686] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 LEADER]: Becoming Leader. State: Replica: 94278d0a47dc47f2a0d5be9589f641be, State: Running, Role: LEADER
I20240503 02:14:10.829422 32706 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 960, Committed index: 960, Last appended: 1.960, Last appended by leader: 960, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:10.833207 27514 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be reported cstate change: term changed from 1 to 2, leader changed from acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) to 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71). New cstate: current_term: 2 leader_uuid: "94278d0a47dc47f2a0d5be9589f641be" committed_config { opid_index: 778 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
W20240503 02:14:10.882761 27745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46266: Illegal state: replica acd475a9eee143a89c1d28a05fa51d26 is not leader of this config: current role FOLLOWER
W20240503 02:14:10.949174 29173 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47526: Illegal state: replica 24868c13491843548e8da6583ef1e9d5 is not leader of this config: current role FOLLOWER
I20240503 02:14:11.031651 29212 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 1 index: 960. Preceding OpId from leader: term: 2 index: 962. (index mismatch)
I20240503 02:14:11.032895   319 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 961, Last known committed idx: 960, Time since last communication: 0.000s
I20240503 02:14:11.041194 27786 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 1 index: 960. Preceding OpId from leader: term: 2 index: 962. (index mismatch)
I20240503 02:14:11.060199   318 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 961, Last known committed idx: 960, Time since last communication: 0.000s
I20240503 02:14:11.203161 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:11.239430 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:11.290570 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:11.292706 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:11.369560 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:11.523027 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:11.564309 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:14:11.815410 28885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:46503, user_credentials={real_user=slave}} blocked reactor thread for 105121us
I20240503 02:14:12.226294 29344 consensus_queue.cc:237] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 986, Committed index: 986, Last appended: 2.986, Last appended by leader: 960, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 987 OBSOLETE_local: false peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:12.236863 29210 raft_consensus.cc:1261] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94278d0a47dc47f2a0d5be9589f641be: Log matching property violated. Preceding OpId in replica: term: 2 index: 986. Preceding OpId from leader: term: 2 index: 987. (index mismatch)
I20240503 02:14:12.237862   319 consensus_queue.cc:1035] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 987, Last known committed idx: 986, Time since last communication: 0.000s
I20240503 02:14:12.244347   319 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be [term 2 LEADER]: Committing config change with OpId 2.987: config changed from index 778 to 987, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 987 OBSOLETE_local: false peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:14:12.248239 29210 raft_consensus.cc:2932] T 4ec65a7b8d0943119b929ae3a9544ec0 P 24868c13491843548e8da6583ef1e9d5 [term 2 FOLLOWER]: Committing config change with OpId 2.987: config changed from index 778 to 987, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 987 OBSOLETE_local: false peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:14:12.265285 27509 catalog_manager.cc:5484] T 4ec65a7b8d0943119b929ae3a9544ec0 P 94278d0a47dc47f2a0d5be9589f641be reported cstate change: config changed from index 778 to 987, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "94278d0a47dc47f2a0d5be9589f641be" committed_config { opid_index: 987 OBSOLETE_local: false peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } }
I20240503 02:14:12.361235 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ec65a7b8d0943119b929ae3a9544ec0 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 987) from {username='slave'} at 127.0.0.1:51446
I20240503 02:14:12.382014   413 tablet_replica.cc:330] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:14:12.383203   413 raft_consensus.cc:2229] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:12.383790   413 raft_consensus.cc:2258] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:12.398881   413 ts_tablet_manager.cc:1875] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:12.406776   413 ts_tablet_manager.cc:1888] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.986
I20240503 02:14:12.407128   413 log.cc:1196] T 4ec65a7b8d0943119b929ae3a9544ec0 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:14:12.410362 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet 4ec65a7b8d0943119b929ae3a9544ec0 (table rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]) successfully deleted
I20240503 02:14:13.219180 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:13.272176 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:13.285981 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:13.364617 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:13.365877 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:13.409538 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:13.548175 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:14:13.814674 27850 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:40065, user_credentials={real_user=slave}} blocked reactor thread for 53700.7us
I20240503 02:14:14.300175 28951 consensus_queue.cc:237] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1110, Committed index: 1110, Last appended: 3.1110, Last appended by leader: 1019, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } }
I20240503 02:14:14.306033 29211 raft_consensus.cc:1261] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 3 index: 1110. Preceding OpId from leader: term: 3 index: 1111. (index mismatch)
I20240503 02:14:14.319841   445 consensus_queue.cc:1035] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1111, Last known committed idx: 1110, Time since last communication: 0.000s
I20240503 02:14:14.327548   445 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef [term 3 LEADER]: Committing config change with OpId 3.1111: config changed from index 860 to 1111, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:14.334818 29211 raft_consensus.cc:2932] T a63be7e6f2da4163baced11894065444 P 24868c13491843548e8da6583ef1e9d5 [term 3 FOLLOWER]: Committing config change with OpId 3.1111: config changed from index 860 to 1111, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:14.350803 27509 catalog_manager.cc:5484] T a63be7e6f2da4163baced11894065444 P b0eb49c52e794992a8fc34173f0cdcef reported cstate change: config changed from index 860 to 1111, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 3 leader_uuid: "b0eb49c52e794992a8fc34173f0cdcef" committed_config { opid_index: 1111 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "24868c13491843548e8da6583ef1e9d5" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 37393 } } }
I20240503 02:14:14.474603 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet a63be7e6f2da4163baced11894065444 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 1111) from {username='slave'} at 127.0.0.1:51446
I20240503 02:14:14.480973   514 tablet_replica.cc:330] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:14:14.481290   514 raft_consensus.cc:2229] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:14.481601   514 raft_consensus.cc:2258] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:14.488148   514 ts_tablet_manager.cc:1875] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:14.513265   514 ts_tablet_manager.cc:1888] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1110
I20240503 02:14:14.525952   514 log.cc:1196] T a63be7e6f2da4163baced11894065444 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/a63be7e6f2da4163baced11894065444
I20240503 02:14:14.527074 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet a63be7e6f2da4163baced11894065444 (table rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]) successfully deleted
I20240503 02:14:15.171080 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:15.195654 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:15.330291 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:15.360082 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:15.371268 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:15.421212 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:15.519026 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:16.589160 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:16.620664 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:16.639283 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:16.668887 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:16.768919 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:16.783675 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:16.872617 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:14:18.175172 29971 log.cc:924] Time spent T 878e847516c74149966db83f6f70c6d0 P 94278d0a47dc47f2a0d5be9589f641be: Append to log took a long time: real 0.056s	user 0.013s	sys 0.000s
I20240503 02:14:19.525240 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:19.698921 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:19.751416 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:19.772670 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:19.806032 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:19.924482 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:20.018613 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:14:20.330483 27592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:32881, user_credentials={real_user=slave}} blocked reactor thread for 43618.1us
I20240503 02:14:21.099529 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:21.163795 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:21.195586 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:21.205837 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:21.219455 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:21.281561 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:21.301854 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:22.202322 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:22.202991 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:22.224318 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:22.245015 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:22.250213 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:22.263983 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:22.265974 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 e729927c47f74970b0a958b67ee29059 | 127.20.218.126:37537 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:39207 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 24868c13491843548e8da6583ef1e9d5 | 127.20.218.70:37393 | HEALTHY | <none>   |       0        |       0
 94278d0a47dc47f2a0d5be9589f641be | 127.20.218.71:40065 | HEALTHY | <none>   |       2        |       0
 a8ef897d344843e2bbbbccc342f924ca | 127.20.218.69:32881 | HEALTHY | <none>   |       0        |       0
 acd475a9eee143a89c1d28a05fa51d26 | 127.20.218.66:46503 | HEALTHY | <none>   |       5        |       0
 b0eb49c52e794992a8fc34173f0cdcef | 127.20.218.68:39319 | HEALTHY | <none>   |       1        |       0
 eaffa047cf9b43b79a5b75089ea63901 | 127.20.218.67:45743 | HEALTHY | <none>   |       3        |       0
 ecf926bd3fb84e0d8f66465d78a70d31 | 127.20.218.65:44115 | HEALTHY | <none>   |       5        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.20.218.65                                                                                                                                                    | experimental          | 127.20.218.65:44115
 local_ip_for_outbound_sockets            | 127.20.218.66                                                                                                                                                    | experimental          | 127.20.218.66:46503
 local_ip_for_outbound_sockets            | 127.20.218.67                                                                                                                                                    | experimental          | 127.20.218.67:45743
 local_ip_for_outbound_sockets            | 127.20.218.68                                                                                                                                                    | experimental          | 127.20.218.68:39319
 local_ip_for_outbound_sockets            | 127.20.218.69                                                                                                                                                    | experimental          | 127.20.218.69:32881
 local_ip_for_outbound_sockets            | 127.20.218.70                                                                                                                                                    | experimental          | 127.20.218.70:37393
 local_ip_for_outbound_sockets            | 127.20.218.71                                                                                                                                                    | experimental          | 127.20.218.71:40065
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.20.218.65:44115
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.20.218.66:46503
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.20.218.67:45743
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.20.218.68:39319
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.20.218.69:32881
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.20.218.70:37393
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.20.218.71:40065

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:39207 | all 7 server(s) checked
 time_source         | builtin             | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:14:22.376489 21353 log_verifier.cc:126] Checking tablet 1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:14:22.857599 21353 log_verifier.cc:177] Verified matching terms for 1276 ops in tablet 1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:14:22.860350 21353 log_verifier.cc:126] Checking tablet 376511df6d8645268134e125791f6aca
I20240503 02:14:23.276569 21353 log_verifier.cc:177] Verified matching terms for 1213 ops in tablet 376511df6d8645268134e125791f6aca
I20240503 02:14:23.278054 21353 log_verifier.cc:126] Checking tablet 42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:14:23.294612 21353 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:14:23.294749 21353 log_verifier.cc:126] Checking tablet 4395a743162549ca96757c5838c0e190
I20240503 02:14:23.697759 21353 log_verifier.cc:177] Verified matching terms for 1175 ops in tablet 4395a743162549ca96757c5838c0e190
I20240503 02:14:23.699867 21353 log_verifier.cc:126] Checking tablet 4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:24.120052 21353 log_verifier.cc:177] Verified matching terms for 1246 ops in tablet 4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:24.122160 21353 log_verifier.cc:126] Checking tablet 4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:14:24.519503 21353 log_verifier.cc:177] Verified matching terms for 1134 ops in tablet 4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:14:24.520824 21353 log_verifier.cc:126] Checking tablet 6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:24.910180 21353 log_verifier.cc:177] Verified matching terms for 1126 ops in tablet 6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:24.911487 21353 log_verifier.cc:126] Checking tablet 6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:14:25.311578 21353 log_verifier.cc:177] Verified matching terms for 1192 ops in tablet 6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:14:25.312912 21353 log_verifier.cc:126] Checking tablet 878e847516c74149966db83f6f70c6d0
I20240503 02:14:25.716712 21353 log_verifier.cc:177] Verified matching terms for 1147 ops in tablet 878e847516c74149966db83f6f70c6d0
I20240503 02:14:25.718087 21353 log_verifier.cc:126] Checking tablet 896ae927d34e46188120e705fb0ae361
I20240503 02:14:26.196053 21353 log_verifier.cc:177] Verified matching terms for 1324 ops in tablet 896ae927d34e46188120e705fb0ae361
I20240503 02:14:26.197619 21353 log_verifier.cc:126] Checking tablet 8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:14:26.629009 21353 log_verifier.cc:177] Verified matching terms for 1263 ops in tablet 8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:14:26.630836 21353 log_verifier.cc:126] Checking tablet 9698b9fc77ea4172b58f410383f2d926
I20240503 02:14:27.083225 21353 log_verifier.cc:177] Verified matching terms for 1162 ops in tablet 9698b9fc77ea4172b58f410383f2d926
I20240503 02:14:27.084553 21353 log_verifier.cc:126] Checking tablet a52a6040dd89493da32af4b283774f03
I20240503 02:14:27.489425 21353 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet a52a6040dd89493da32af4b283774f03
I20240503 02:14:27.490779 21353 log_verifier.cc:126] Checking tablet a63be7e6f2da4163baced11894065444
I20240503 02:14:27.946192 21353 log_verifier.cc:177] Verified matching terms for 1232 ops in tablet a63be7e6f2da4163baced11894065444
I20240503 02:14:27.947672 21353 log_verifier.cc:126] Checking tablet ddd409a1c4554665831eceef070ab660
I20240503 02:14:28.356832 21353 log_verifier.cc:177] Verified matching terms for 1213 ops in tablet ddd409a1c4554665831eceef070ab660
I20240503 02:14:28.358419 21353 log_verifier.cc:126] Checking tablet f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:14:28.792816 21353 log_verifier.cc:177] Verified matching terms for 1154 ops in tablet f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:14:28.804916 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.72:0
--local_ip_for_outbound_sockets=127.20.218.72
--webserver_interface=127.20.218.72
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--builtin_ntp_servers=127.20.218.84:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:14:29.217790   843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:29.218125   843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:29.218217   843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:29.226198   843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:29.226342   843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:29.226459   843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.72
I20240503 02:14:29.237170   843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.218.72
--webserver_port=0
--tserver_master_addrs=127.20.218.126:37537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.72
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:29.238571   843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:29.240494   843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:29.254962   849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:29.255064   851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:29.255856   848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:29.256103   843 server_base.cc:1034] running on GCE node
I20240503 02:14:29.256808   843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:29.257988   843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:29.259245   843 hybrid_clock.cc:648] HybridClock initialized: now 1714702469259200 us; error 55 us; skew 500 ppm
I20240503 02:14:29.259701   843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:29.262676   843 webserver.cc:460] Webserver started at http://127.20.218.72:37061/ using document root <none> and password file <none>
I20240503 02:14:29.263598   843 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:29.263708   843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:29.264051   843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:29.267052   843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "2ce83eb95475412c867e429ed43f0e65"
format_stamp: "Formatted at 2024-05-03 02:14:29 on dist-test-slave-81lq"
I20240503 02:14:29.267948   843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "2ce83eb95475412c867e429ed43f0e65"
format_stamp: "Formatted at 2024-05-03 02:14:29 on dist-test-slave-81lq"
I20240503 02:14:29.274691   843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240503 02:14:29.280303   857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:29.282459   843 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:14:29.282857   843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2ce83eb95475412c867e429ed43f0e65"
format_stamp: "Formatted at 2024-05-03 02:14:29 on dist-test-slave-81lq"
I20240503 02:14:29.283186   843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:29.320106   843 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:29.321132   843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:29.321478   843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:29.322682   843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:29.324268   843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:29.324358   843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:29.324472   843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:29.324570   843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:29.368029   843 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.72:39621
I20240503 02:14:29.368090   969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.72:39621 every 8 connection(s)
I20240503 02:14:29.369599   843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:14:29.375759 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 843
I20240503 02:14:29.375979 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:14:29.397253   970 heartbeater.cc:347] Connected to a master server at 127.20.218.126:37537
I20240503 02:14:29.397647   970 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:29.398571   970 heartbeater.cc:511] Master 127.20.218.126:37537 requested a full tablet report, sending...
I20240503 02:14:29.400164 27511 ts_manager.cc:194] Registered new tserver with Master: 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621)
I20240503 02:14:29.401010 27511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.72:35289
I20240503 02:14:29.969419 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:29.972622   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.000777 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:30.007066 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.024638 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:30.038345 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:30.104951 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:30.126880 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:30.314978 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.321041   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.325206 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:30.377010 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:30.382532 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.387127 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:30.402196 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:30.404453   970 heartbeater.cc:503] Master 127.20.218.126:37537 was elected leader, sending a full tablet report...
I20240503 02:14:30.409961 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:30.556154 27654 consensus_queue.cc:237] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1126, Committed index: 1126, Last appended: 1.1126, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:30.559360 29341 raft_consensus.cc:1261] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 1126. Preceding OpId from leader: term: 1 index: 1127. (index mismatch)
I20240503 02:14:30.560055  1070 consensus_queue.cc:1035] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1127, Last known committed idx: 1126, Time since last communication: 0.000s
I20240503 02:14:30.563257   667 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.1127: config changed from index 207 to 1127, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.563988 29341 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Committing config change with OpId 1.1127: config changed from index 207 to 1127, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.569424 27514 catalog_manager.cc:5484] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 207 to 1127, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
W20240503 02:14:30.570811 27590 consensus_peers.cc:485] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6af88acfe24e49279a98b5dcf417bed7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:30.614156 27786 consensus_queue.cc:237] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1246, Committed index: 1246, Last appended: 1.1246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:30.617198 27914 raft_consensus.cc:1261] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Refusing update from remote peer acd475a9eee143a89c1d28a05fa51d26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1246. Preceding OpId from leader: term: 1 index: 1247. (index mismatch)
I20240503 02:14:30.618011 32383 consensus_queue.cc:1035] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1247, Last known committed idx: 1246, Time since last communication: 0.000s
W20240503 02:14:30.623253 27721 consensus_peers.cc:485] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dd7e1c1b87f4f6db714735a13e25f67. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:30.622974  1093 raft_consensus.cc:2932] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Committing config change with OpId 1.1247: config changed from index -1 to 1247, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.623847 27914 raft_consensus.cc:2932] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Committing config change with OpId 1.1247: config changed from index -1 to 1247, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.627856 27509 catalog_manager.cc:5484] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 reported cstate change: config changed from index -1 to 1247, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "acd475a9eee143a89c1d28a05fa51d26" committed_config { opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.664335 27654 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1324, Committed index: 1324, Last appended: 1.1324, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:30.666508 29079 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 1324. Preceding OpId from leader: term: 1 index: 1325. (index mismatch)
W20240503 02:14:30.666674 27590 consensus_peers.cc:485] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 896ae927d34e46188120e705fb0ae361. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:30.667018  1070 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1325, Last known committed idx: 1324, Time since last communication: 0.000s
I20240503 02:14:30.669907   667 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.1325: config changed from index 574 to 1325, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.673302 29079 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Committing config change with OpId 1.1325: config changed from index 574 to 1325, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.674674 27511 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 574 to 1325, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.719560 27914 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1175, Committed index: 1175, Last appended: 2.1175, Last appended by leader: 647, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:30.722638 28949 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 2 index: 1175. Preceding OpId from leader: term: 2 index: 1176. (index mismatch)
I20240503 02:14:30.723749  1130 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1176, Last known committed idx: 1175, Time since last communication: 0.000s
I20240503 02:14:30.727924   832 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Committing config change with OpId 2.1176: config changed from index 770 to 1176, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.728796 28949 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Committing config change with OpId 2.1176: config changed from index 770 to 1176, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New config: { opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:30.734234 27511 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef reported cstate change: config changed from index 770 to 1176, VOTER 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72) added. New cstate: current_term: 2 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
W20240503 02:14:30.736120 27851 consensus_peers.cc:485] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4395a743162549ca96757c5838c0e190. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:14:30.874027 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.894392   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.912427 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:30.918855 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:30.945366 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:30.949587 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:30.988987 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:31.005856 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:31.040865  1174 ts_tablet_manager.cc:902] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Initiating tablet copy from peer acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503)
I20240503 02:14:31.046919  1174 tablet_copy_client.cc:308] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:46503
I20240503 02:14:31.055219 27807 tablet_copy_service.cc:140] P acd475a9eee143a89c1d28a05fa51d26: Received BeginTabletCopySession request for tablet 4dd7e1c1b87f4f6db714735a13e25f67 from peer 2ce83eb95475412c867e429ed43f0e65 ({username='slave'} at 127.20.218.72:51021)
I20240503 02:14:31.055455 27807 tablet_copy_service.cc:161] P acd475a9eee143a89c1d28a05fa51d26: Beginning new tablet copy session on tablet 4dd7e1c1b87f4f6db714735a13e25f67 from peer 2ce83eb95475412c867e429ed43f0e65 at {username='slave'} at 127.20.218.72:51021: session id = 2ce83eb95475412c867e429ed43f0e65-4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:31.057756 27807 tablet_copy_source_session.cc:205] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:31.063359  1174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dd7e1c1b87f4f6db714735a13e25f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:31.078711  1174 tablet_copy_client.cc:783] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:31.079244  1174 tablet_copy_client.cc:647] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:31.085588  1174 tablet_copy_client.cc:523] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:31.090021  1174 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap starting.
I20240503 02:14:31.131704  1186 ts_tablet_manager.cc:902] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:14:31.134335  1186 tablet_copy_client.cc:308] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:14:31.142081 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 6af88acfe24e49279a98b5dcf417bed7 from peer 2ce83eb95475412c867e429ed43f0e65 ({username='slave'} at 127.20.218.72:40541)
I20240503 02:14:31.142282 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 6af88acfe24e49279a98b5dcf417bed7 from peer 2ce83eb95475412c867e429ed43f0e65 at {username='slave'} at 127.20.218.72:40541: session id = 2ce83eb95475412c867e429ed43f0e65-6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:31.146118 27676 tablet_copy_source_session.cc:205] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:31.148056  1186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af88acfe24e49279a98b5dcf417bed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:31.153854  1186 tablet_copy_client.cc:783] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:31.154172  1186 tablet_copy_client.cc:647] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:31.159435  1186 tablet_copy_client.cc:523] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:31.162626  1186 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap starting.
I20240503 02:14:31.196341  1204 ts_tablet_manager.cc:902] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Initiating tablet copy from peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
I20240503 02:14:31.242815  1204 tablet_copy_client.cc:308] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:44115
I20240503 02:14:31.243836 27676 tablet_copy_service.cc:140] P ecf926bd3fb84e0d8f66465d78a70d31: Received BeginTabletCopySession request for tablet 896ae927d34e46188120e705fb0ae361 from peer 2ce83eb95475412c867e429ed43f0e65 ({username='slave'} at 127.20.218.72:40541)
I20240503 02:14:31.244025 27676 tablet_copy_service.cc:161] P ecf926bd3fb84e0d8f66465d78a70d31: Beginning new tablet copy session on tablet 896ae927d34e46188120e705fb0ae361 from peer 2ce83eb95475412c867e429ed43f0e65 at {username='slave'} at 127.20.218.72:40541: session id = 2ce83eb95475412c867e429ed43f0e65-896ae927d34e46188120e705fb0ae361
I20240503 02:14:31.247232 27676 tablet_copy_source_session.cc:205] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:31.251546  1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896ae927d34e46188120e705fb0ae361. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:31.254477 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:31.274351   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:31.274947  1213 ts_tablet_manager.cc:902] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Initiating tablet copy from peer eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743)
I20240503 02:14:31.276181  1204 tablet_copy_client.cc:783] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:31.277665  1204 tablet_copy_client.cc:647] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:31.289453  1204 tablet_copy_client.cc:523] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:31.293310  1204 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap starting.
I20240503 02:14:31.322768  1213 tablet_copy_client.cc:308] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:45743
I20240503 02:14:31.324551 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:31.325065 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:31.325510 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:31.339501 27937 tablet_copy_service.cc:140] P eaffa047cf9b43b79a5b75089ea63901: Received BeginTabletCopySession request for tablet 4395a743162549ca96757c5838c0e190 from peer 2ce83eb95475412c867e429ed43f0e65 ({username='slave'} at 127.20.218.72:48801)
I20240503 02:14:31.339740 27937 tablet_copy_service.cc:161] P eaffa047cf9b43b79a5b75089ea63901: Beginning new tablet copy session on tablet 4395a743162549ca96757c5838c0e190 from peer 2ce83eb95475412c867e429ed43f0e65 at {username='slave'} at 127.20.218.72:48801: session id = 2ce83eb95475412c867e429ed43f0e65-4395a743162549ca96757c5838c0e190
I20240503 02:14:31.343138 27937 tablet_copy_source_session.cc:205] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:14:31.344034 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:31.345866  1213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4395a743162549ca96757c5838c0e190. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:31.347517 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:31.378343 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:31.419343  1213 tablet_copy_client.cc:783] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 0 data blocks...
I20240503 02:14:31.419680  1213 tablet_copy_client.cc:647] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Starting download of 1 WAL segments...
I20240503 02:14:31.428536  1213 tablet_copy_client.cc:523] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:14:31.448565  1213 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap starting.
I20240503 02:14:31.555018  1174 log.cc:824] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:32.047053 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.101455 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:32.112417 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.133769   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.135425 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:32.155244 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:32.197875 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:32.227133 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:32.736630 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.749713   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.760836 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:32.788203 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:32.807147 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:32.817757 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:32.852506 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:32.875151 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:33.422639 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:33.448205   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:33.502985 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:33.504633 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:33.539047 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:33.539532 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:33.558377 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:33.563582 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:14:33.645283 27590 consensus_peers.cc:485] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:33.698508 27851 consensus_peers.cc:485] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:33.779686 27721 consensus_peers.cc:485] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240503 02:14:33.845678 27590 consensus_peers.cc:485] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 -> Peer 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621): Couldn't send request to peer 2ce83eb95475412c867e429ed43f0e65. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240503 02:14:33.878616  1186 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=1127 overwritten=0 applied=1127 ignored=0} inserts{seen=1810 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:33.888499  1186 tablet_bootstrap.cc:492] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap complete.
I20240503 02:14:33.893234  1186 ts_tablet_manager.cc:1367] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Time spent bootstrapping tablet: real 2.731s	user 1.346s	sys 0.104s
I20240503 02:14:33.901880  1186 raft_consensus.cc:348] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:33.904600  1186 raft_consensus.cc:729] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce83eb95475412c867e429ed43f0e65, State: Initialized, Role: FOLLOWER
I20240503 02:14:33.905591  1186 consensus_queue.cc:260] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1127, Last appended: 1.1127, Last appended by leader: 1127, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1127 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:33.923139  1186 ts_tablet_manager.cc:1398] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65: Time spent starting tablet: real 0.030s	user 0.000s	sys 0.024s
I20240503 02:14:33.924252 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 2ce83eb95475412c867e429ed43f0e65-6af88acfe24e49279a98b5dcf417bed7 received from {username='slave'} at 127.20.218.72:40541
I20240503 02:14:33.924389 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 2ce83eb95475412c867e429ed43f0e65-6af88acfe24e49279a98b5dcf417bed7 on tablet 6af88acfe24e49279a98b5dcf417bed7 with peer 2ce83eb95475412c867e429ed43f0e65
I20240503 02:14:34.065836  1213 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=1176 overwritten=0 applied=1176 ignored=0} inserts{seen=1845 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:34.067509  1213 tablet_bootstrap.cc:492] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap complete.
I20240503 02:14:34.068060  1213 ts_tablet_manager.cc:1367] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Time spent bootstrapping tablet: real 2.620s	user 1.423s	sys 0.163s
I20240503 02:14:34.071369  1213 raft_consensus.cc:348] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.071660  1213 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce83eb95475412c867e429ed43f0e65, State: Initialized, Role: FOLLOWER
I20240503 02:14:34.071935  1213 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1176, Last appended: 2.1176, Last appended by leader: 1176, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.072526  1213 ts_tablet_manager.cc:1398] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.000s
I20240503 02:14:34.074030 27937 tablet_copy_service.cc:342] P eaffa047cf9b43b79a5b75089ea63901: Request end of tablet copy session 2ce83eb95475412c867e429ed43f0e65-4395a743162549ca96757c5838c0e190 received from {username='slave'} at 127.20.218.72:48801
I20240503 02:14:34.074152 27937 tablet_copy_service.cc:433] P eaffa047cf9b43b79a5b75089ea63901: ending tablet copy session 2ce83eb95475412c867e429ed43f0e65-4395a743162549ca96757c5838c0e190 on tablet 4395a743162549ca96757c5838c0e190 with peer 2ce83eb95475412c867e429ed43f0e65
I20240503 02:14:34.118072 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.136142   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.152423   925 raft_consensus.cc:1203] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1126->[1.1127-1.1127]   Dedup: 1.1127->[]
I20240503 02:14:34.172075 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:34.212805 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:34.223285   925 raft_consensus.cc:1203] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1175->[2.1176-2.1176]   Dedup: 2.1176->[]
I20240503 02:14:34.260951 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:34.283902  1174 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=1247 overwritten=0 applied=1247 ignored=0} inserts{seen=1878 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:34.288201  1174 tablet_bootstrap.cc:492] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap complete.
I20240503 02:14:34.288697  1174 ts_tablet_manager.cc:1367] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Time spent bootstrapping tablet: real 3.199s	user 1.499s	sys 0.134s
I20240503 02:14:34.290113  1174 raft_consensus.cc:348] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.290385  1174 raft_consensus.cc:729] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce83eb95475412c867e429ed43f0e65, State: Initialized, Role: FOLLOWER
I20240503 02:14:34.291224  1174 consensus_queue.cc:260] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1247, Last appended: 1.1247, Last appended by leader: 1247, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.291844  1174 ts_tablet_manager.cc:1398] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:14:34.292979 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:34.294365 27807 tablet_copy_service.cc:342] P acd475a9eee143a89c1d28a05fa51d26: Request end of tablet copy session 2ce83eb95475412c867e429ed43f0e65-4dd7e1c1b87f4f6db714735a13e25f67 received from {username='slave'} at 127.20.218.72:51021
I20240503 02:14:34.294508 27807 tablet_copy_service.cc:433] P acd475a9eee143a89c1d28a05fa51d26: ending tablet copy session 2ce83eb95475412c867e429ed43f0e65-4dd7e1c1b87f4f6db714735a13e25f67 on tablet 4dd7e1c1b87f4f6db714735a13e25f67 with peer 2ce83eb95475412c867e429ed43f0e65
I20240503 02:14:34.306418 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.319278 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:34.492545  1204 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=1325 overwritten=0 applied=1325 ignored=0} inserts{seen=1995 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:14:34.492961  1204 tablet_bootstrap.cc:492] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Bootstrap complete.
I20240503 02:14:34.493362  1204 ts_tablet_manager.cc:1367] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Time spent bootstrapping tablet: real 3.200s	user 1.550s	sys 0.194s
I20240503 02:14:34.494171  1204 raft_consensus.cc:348] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.494443  1204 raft_consensus.cc:729] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce83eb95475412c867e429ed43f0e65, State: Initialized, Role: FOLLOWER
I20240503 02:14:34.494730  1204 consensus_queue.cc:260] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1325, Last appended: 1.1325, Last appended by leader: 1325, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:34.495275  1204 ts_tablet_manager.cc:1398] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:14:34.496050 27676 tablet_copy_service.cc:342] P ecf926bd3fb84e0d8f66465d78a70d31: Request end of tablet copy session 2ce83eb95475412c867e429ed43f0e65-896ae927d34e46188120e705fb0ae361 received from {username='slave'} at 127.20.218.72:40541
I20240503 02:14:34.496157 27676 tablet_copy_service.cc:433] P ecf926bd3fb84e0d8f66465d78a70d31: ending tablet copy session 2ce83eb95475412c867e429ed43f0e65-896ae927d34e46188120e705fb0ae361 on tablet 896ae927d34e46188120e705fb0ae361 with peer 2ce83eb95475412c867e429ed43f0e65
I20240503 02:14:34.551756 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.562724   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.567284 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:34.580906 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:34.581056 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:34.584805 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:34.617753 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:34.643545 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:34.772253   925 raft_consensus.cc:1203] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1246->[1.1247-1.1247]   Dedup: 1.1247->[]
I20240503 02:14:34.975360   925 raft_consensus.cc:1203] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1324->[1.1325-1.1325]   Dedup: 1.1325->[]
I20240503 02:14:35.047724   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.072402 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.079001 28932 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:35.085599 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.088791 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.097270 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.106716 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:35.121667 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.211717 27914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4395a743162549ca96757c5838c0e190"
dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40280
I20240503 02:14:35.211997 27914 raft_consensus.cc:595] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:14:35.316452  1505 raft_consensus.cc:979] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: : Instructing follower b0eb49c52e794992a8fc34173f0cdcef to start an election
I20240503 02:14:35.316689  1505 raft_consensus.cc:1067] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Signalling peer b0eb49c52e794992a8fc34173f0cdcef to start an election
I20240503 02:14:35.317662 28949 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4395a743162549ca96757c5838c0e190"
dest_uuid: "b0eb49c52e794992a8fc34173f0cdcef"
 from {username='slave'} at 127.20.218.67:41649
I20240503 02:14:35.317895 28949 raft_consensus.cc:482] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:35.318009 28949 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:35.320849 28949 raft_consensus.cc:504] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.323276 28949 leader_election.cc:288] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [CANDIDATE]: Term 3 election: Requested vote from peers eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743), 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621)
I20240503 02:14:35.333464   925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "b0eb49c52e794992a8fc34173f0cdcef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1176 } } ignore_live_leader: true dest_uuid: "2ce83eb95475412c867e429ed43f0e65"
I20240503 02:14:35.333889   925 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:35.337109   925 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0eb49c52e794992a8fc34173f0cdcef in term 3.
I20240503 02:14:35.339159 27914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4395a743162549ca96757c5838c0e190" candidate_uuid: "b0eb49c52e794992a8fc34173f0cdcef" candidate_term: 3 candidate_status { last_received { term: 2 index: 1176 } } ignore_live_leader: true dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
I20240503 02:14:35.339391 27914 raft_consensus.cc:3032] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:14:35.339476 27914 raft_consensus.cc:729] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:14:35.339742 27914 consensus_queue.cc:260] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1176, Committed index: 1176, Last appended: 2.1176, Last appended by leader: 1176, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.340126 27914 raft_consensus.cc:3037] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:14:35.343622 27914 raft_consensus.cc:2454] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0eb49c52e794992a8fc34173f0cdcef in term 3.
I20240503 02:14:35.344359 28888 leader_election.cc:302] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0eb49c52e794992a8fc34173f0cdcef, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:14:35.344949  1478 raft_consensus.cc:2781] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:14:35.345155  1478 raft_consensus.cc:686] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 3 LEADER]: Becoming Leader. State: Replica: b0eb49c52e794992a8fc34173f0cdcef, State: Running, Role: LEADER
I20240503 02:14:35.345542  1478 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1176, Committed index: 1176, Last appended: 2.1176, Last appended by leader: 1176, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.353744 27514 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef reported cstate change: term changed from 2 to 3, leader changed from eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) to b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68). New cstate: current_term: 3 leader_uuid: "b0eb49c52e794992a8fc34173f0cdcef" committed_config { opid_index: 1176 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:35.362247 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.390328   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.392794 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.394392 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.397727 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.418237 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.492702 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.502157 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.713344 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.713793   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.726733 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.728152 29063 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:35.737928 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.753543 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.775604 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:35.804339 27635 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:35.805568 27914 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1176. Preceding OpId from leader: term: 3 index: 1177. (index mismatch)
I20240503 02:14:35.806231  1478 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1177, Last known committed idx: 1176, Time since last communication: 0.000s
I20240503 02:14:35.859692   924 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 2 index: 1176. Preceding OpId from leader: term: 3 index: 1177. (index mismatch)
I20240503 02:14:35.860769  1476 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1177, Last known committed idx: 1176, Time since last communication: 0.000s
I20240503 02:14:35.928365 27654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "896ae927d34e46188120e705fb0ae361"
dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33806
I20240503 02:14:35.928593 27654 raft_consensus.cc:595] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:35.944824  1576 raft_consensus.cc:979] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: : Instructing follower a8ef897d344843e2bbbbccc342f924ca to start an election
I20240503 02:14:35.945001  1576 raft_consensus.cc:1067] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Signalling peer a8ef897d344843e2bbbbccc342f924ca to start an election
I20240503 02:14:35.945819 29079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "896ae927d34e46188120e705fb0ae361"
dest_uuid: "a8ef897d344843e2bbbbccc342f924ca"
 from {username='slave'} at 127.20.218.65:41021
I20240503 02:14:35.946142 29079 raft_consensus.cc:482] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:35.946384 29079 raft_consensus.cc:3037] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:35.949461 29079 raft_consensus.cc:504] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.951227 29079 leader_election.cc:288] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [CANDIDATE]: Term 2 election: Requested vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115), 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621)
I20240503 02:14:35.959292   924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896ae927d34e46188120e705fb0ae361" candidate_uuid: "a8ef897d344843e2bbbbccc342f924ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1325 } } ignore_live_leader: true dest_uuid: "2ce83eb95475412c867e429ed43f0e65"
I20240503 02:14:35.959604   924 raft_consensus.cc:3037] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:35.962679   924 raft_consensus.cc:2454] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ef897d344843e2bbbbccc342f924ca in term 2.
I20240503 02:14:35.963347 29017 leader_election.cc:302] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce83eb95475412c867e429ed43f0e65, a8ef897d344843e2bbbbccc342f924ca; no voters: 
I20240503 02:14:35.963672 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896ae927d34e46188120e705fb0ae361" candidate_uuid: "a8ef897d344843e2bbbbccc342f924ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1325 } } ignore_live_leader: true dest_uuid: "ecf926bd3fb84e0d8f66465d78a70d31"
I20240503 02:14:35.963893 27654 raft_consensus.cc:3032] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:35.963979 27654 raft_consensus.cc:729] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecf926bd3fb84e0d8f66465d78a70d31, State: Running, Role: LEADER
I20240503 02:14:35.964335  1579 raft_consensus.cc:2781] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:35.964363 27654 consensus_queue.cc:260] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1325, Committed index: 1325, Last appended: 1.1325, Last appended by leader: 1325, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.964844 27654 raft_consensus.cc:3037] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:35.965575  1579 raft_consensus.cc:686] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 2 LEADER]: Becoming Leader. State: Replica: a8ef897d344843e2bbbbccc342f924ca, State: Running, Role: LEADER
I20240503 02:14:35.966123  1579 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1325, Committed index: 1325, Last appended: 1.1325, Last appended by leader: 1325, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:35.970764 27654 raft_consensus.cc:2454] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ef897d344843e2bbbbccc342f924ca in term 2.
I20240503 02:14:35.971416 27510 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca reported cstate change: term changed from 1 to 2, leader changed from ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) to a8ef897d344843e2bbbbccc342f924ca (127.20.218.69). New cstate: current_term: 2 leader_uuid: "a8ef897d344843e2bbbbccc342f924ca" committed_config { opid_index: 1325 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:36.038646 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.048064   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.050937 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.066154 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:36.068387 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.069274 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:36.102648 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.169083 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:36.348621 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.352821   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.360855 27654 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Refusing update from remote peer a8ef897d344843e2bbbbccc342f924ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1325. Preceding OpId from leader: term: 2 index: 1326. (index mismatch)
I20240503 02:14:36.365362  1579 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1326, Last known committed idx: 1325, Time since last communication: 0.000s
I20240503 02:14:36.385933 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:36.398547   924 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Refusing update from remote peer a8ef897d344843e2bbbbccc342f924ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 1325. Preceding OpId from leader: term: 2 index: 1326. (index mismatch)
I20240503 02:14:36.399032 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.399623  1579 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1326, Last known committed idx: 1325, Time since last communication: 0.000s
I20240503 02:14:36.446417 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:36.482664 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.530514 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.536037 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:36.641067 27655 consensus_queue.cc:237] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1127, Committed index: 1127, Last appended: 1.1127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1128 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:36.643556   925 raft_consensus.cc:1261] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Refusing update from remote peer ecf926bd3fb84e0d8f66465d78a70d31: Log matching property violated. Preceding OpId in replica: term: 1 index: 1127. Preceding OpId from leader: term: 1 index: 1128. (index mismatch)
I20240503 02:14:36.644228  1576 consensus_queue.cc:1035] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1128, Last known committed idx: 1127, Time since last communication: 0.000s
I20240503 02:14:36.648608  1669 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 [term 1 LEADER]: Committing config change with OpId 1.1128: config changed from index 1127 to 1128, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) evicted. New config: { opid_index: 1128 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:36.649556   925 raft_consensus.cc:2932] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Committing config change with OpId 1.1128: config changed from index 1127 to 1128, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) evicted. New config: { opid_index: 1128 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:36.654130 27510 catalog_manager.cc:5484] T 6af88acfe24e49279a98b5dcf417bed7 P ecf926bd3fb84e0d8f66465d78a70d31 reported cstate change: config changed from index 1127 to 1128, VOTER 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71) evicted. New cstate: current_term: 1 leader_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" committed_config { opid_index: 1128 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:36.665952 29324 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af88acfe24e49279a98b5dcf417bed7 with delete_type TABLET_DATA_TOMBSTONED (TS 94278d0a47dc47f2a0d5be9589f641be not found in new config with opid_index 1128) from {username='slave'} at 127.0.0.1:59830
I20240503 02:14:36.667085  1679 tablet_replica.cc:330] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: stopping tablet replica
I20240503 02:14:36.667629  1679 raft_consensus.cc:2229] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:36.667945  1679 raft_consensus.cc:2258] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:36.670183  1679 ts_tablet_manager.cc:1875] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:36.676208  1679 ts_tablet_manager.cc:1888] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1127
I20240503 02:14:36.676664  1679 log.cc:1196] T 6af88acfe24e49279a98b5dcf417bed7 P 94278d0a47dc47f2a0d5be9589f641be: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/wals/6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:36.678051 27499 catalog_manager.cc:4830] TS 94278d0a47dc47f2a0d5be9589f641be (127.20.218.71:40065): tablet 6af88acfe24e49279a98b5dcf417bed7 (table rebalance_test_table_4 [id=1397d007101b4cf6a7622c4a8aa5841c]) successfully deleted
I20240503 02:14:36.792634   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.795372 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:36.800487 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:36.805527 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:36.809346 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.825407 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.846297 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:36.893921 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:36.969164 27786 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67"
dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46434
I20240503 02:14:36.969472 27786 raft_consensus.cc:595] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:14:37.092778 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:37.110487 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:37.110947   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:37.126087 27766 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:14:37.143100 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:37.160305 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:37.161724 27895 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:37.177415 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:37.328548  1749 raft_consensus.cc:979] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: : Instructing follower eaffa047cf9b43b79a5b75089ea63901 to start an election
I20240503 02:14:37.328745  1749 raft_consensus.cc:1067] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Signalling peer eaffa047cf9b43b79a5b75089ea63901 to start an election
I20240503 02:14:37.329682 27915 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67"
dest_uuid: "eaffa047cf9b43b79a5b75089ea63901"
 from {username='slave'} at 127.20.218.66:46803
I20240503 02:14:37.329906 27915 raft_consensus.cc:482] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:14:37.330008 27915 raft_consensus.cc:3037] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:37.332347 27915 raft_consensus.cc:504] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:37.333370 27915 leader_election.cc:288] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 2 election: Requested vote from peers acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503), 2ce83eb95475412c867e429ed43f0e65 (127.20.218.72:39621)
I20240503 02:14:37.334007 27786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 2 candidate_status { last_received { term: 1 index: 1247 } } ignore_live_leader: true dest_uuid: "acd475a9eee143a89c1d28a05fa51d26"
I20240503 02:14:37.334270 27786 raft_consensus.cc:3032] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:14:37.334220   925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dd7e1c1b87f4f6db714735a13e25f67" candidate_uuid: "eaffa047cf9b43b79a5b75089ea63901" candidate_term: 2 candidate_status { last_received { term: 1 index: 1247 } } ignore_live_leader: true dest_uuid: "2ce83eb95475412c867e429ed43f0e65"
I20240503 02:14:37.334364 27786 raft_consensus.cc:729] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: acd475a9eee143a89c1d28a05fa51d26, State: Running, Role: LEADER
I20240503 02:14:37.334542   925 raft_consensus.cc:3037] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:37.334676 27786 consensus_queue.cc:260] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1247, Committed index: 1247, Last appended: 1.1247, Last appended by leader: 1247, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:37.335072 27786 raft_consensus.cc:3037] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:14:37.337373   925 raft_consensus.cc:2454] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 2.
I20240503 02:14:37.338186 27851 leader_election.cc:302] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce83eb95475412c867e429ed43f0e65, eaffa047cf9b43b79a5b75089ea63901; no voters: 
I20240503 02:14:37.338555 27786 raft_consensus.cc:2454] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaffa047cf9b43b79a5b75089ea63901 in term 2.
I20240503 02:14:37.338621  1505 raft_consensus.cc:2781] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:14:37.339150  1505 raft_consensus.cc:686] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Becoming Leader. State: Replica: eaffa047cf9b43b79a5b75089ea63901, State: Running, Role: LEADER
I20240503 02:14:37.339522  1505 consensus_queue.cc:237] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1247, Committed index: 1247, Last appended: 1.1247, Last appended by leader: 1247, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:37.342376 27510 catalog_manager.cc:5484] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: term changed from 1 to 2, leader changed from acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) to eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 1247 OBSOLETE_local: false peers { permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 } } peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:37.589717 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:37.591851 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:37.602958   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:37.614390 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:37.615926 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:37.624797 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:37.627003 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:37.658406 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:37.783454 28949 consensus_queue.cc:237] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1177, Committed index: 1177, Last appended: 3.1177, Last appended by leader: 1176, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1178 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:37.785537   925 raft_consensus.cc:1261] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Refusing update from remote peer b0eb49c52e794992a8fc34173f0cdcef: Log matching property violated. Preceding OpId in replica: term: 3 index: 1177. Preceding OpId from leader: term: 3 index: 1178. (index mismatch)
I20240503 02:14:37.786136  1476 consensus_queue.cc:1035] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1178, Last known committed idx: 1177, Time since last communication: 0.000s
I20240503 02:14:37.790807  1810 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef [term 3 LEADER]: Committing config change with OpId 3.1178: config changed from index 1176 to 1178, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 1178 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:37.791836   925 raft_consensus.cc:2932] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Committing config change with OpId 3.1178: config changed from index 1176 to 1178, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New config: { opid_index: 1178 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:37.797557 27514 catalog_manager.cc:5484] T 4395a743162549ca96757c5838c0e190 P b0eb49c52e794992a8fc34173f0cdcef reported cstate change: config changed from index 1176 to 1178, VOTER eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67) evicted. New cstate: current_term: 3 leader_uuid: "b0eb49c52e794992a8fc34173f0cdcef" committed_config { opid_index: 1178 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:37.802212 27895 tablet_service.cc:1514] Processing DeleteTablet for tablet 4395a743162549ca96757c5838c0e190 with delete_type TABLET_DATA_TOMBSTONED (TS eaffa047cf9b43b79a5b75089ea63901 not found in new config with opid_index 1178) from {username='slave'} at 127.0.0.1:40704
I20240503 02:14:37.802906  1817 tablet_replica.cc:330] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: stopping tablet replica
I20240503 02:14:37.803155  1817 raft_consensus.cc:2229] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:37.803373  1817 raft_consensus.cc:2258] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:37.805008  1817 ts_tablet_manager.cc:1875] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:37.811812  1817 ts_tablet_manager.cc:1888] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1177
I20240503 02:14:37.812026  1817 log.cc:1196] T 4395a743162549ca96757c5838c0e190 P eaffa047cf9b43b79a5b75089ea63901: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/4395a743162549ca96757c5838c0e190
I20240503 02:14:37.813095 27502 catalog_manager.cc:4830] TS eaffa047cf9b43b79a5b75089ea63901 (127.20.218.67:45743): tablet 4395a743162549ca96757c5838c0e190 (table rebalance_test_table_1 [id=a6d317e9ea5a47709c55d76dc3342b09]) successfully deleted
I20240503 02:14:37.896822 27786 raft_consensus.cc:1261] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 1247. Preceding OpId from leader: term: 2 index: 1248. (index mismatch)
I20240503 02:14:37.898314  1505 consensus_queue.cc:1035] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd475a9eee143a89c1d28a05fa51d26" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 46503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1248, Last known committed idx: 1247, Time since last communication: 0.000s
I20240503 02:14:37.900483   925 raft_consensus.cc:1261] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 1 index: 1247. Preceding OpId from leader: term: 2 index: 1248. (index mismatch)
I20240503 02:14:37.915078  1840 consensus_queue.cc:1035] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1248, Last known committed idx: 1247, Time since last communication: 0.000s
I20240503 02:14:37.950017 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:37.974299   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:38.000065 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.014245 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:38.043381 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:38.070844 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.087298 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:38.096050 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:38.198136 29079 consensus_queue.cc:237] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1326, Committed index: 1326, Last appended: 2.1326, Last appended by leader: 1325, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1327 OBSOLETE_local: false peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:38.200264   924 raft_consensus.cc:1261] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Refusing update from remote peer a8ef897d344843e2bbbbccc342f924ca: Log matching property violated. Preceding OpId in replica: term: 2 index: 1326. Preceding OpId from leader: term: 2 index: 1327. (index mismatch)
I20240503 02:14:38.200904  1580 consensus_queue.cc:1035] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1327, Last known committed idx: 1326, Time since last communication: 0.000s
I20240503 02:14:38.204447  1871 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca [term 2 LEADER]: Committing config change with OpId 2.1327: config changed from index 1325 to 1327, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 1327 OBSOLETE_local: false peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.205367   924 raft_consensus.cc:2932] T 896ae927d34e46188120e705fb0ae361 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Committing config change with OpId 2.1327: config changed from index 1325 to 1327, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New config: { opid_index: 1327 OBSOLETE_local: false peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.209946 27511 catalog_manager.cc:5484] T 896ae927d34e46188120e705fb0ae361 P a8ef897d344843e2bbbbccc342f924ca reported cstate change: config changed from index 1325 to 1327, VOTER ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "a8ef897d344843e2bbbbccc342f924ca" committed_config { opid_index: 1327 OBSOLETE_local: false peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.215225 27635 tablet_service.cc:1514] Processing DeleteTablet for tablet 896ae927d34e46188120e705fb0ae361 with delete_type TABLET_DATA_TOMBSTONED (TS ecf926bd3fb84e0d8f66465d78a70d31 not found in new config with opid_index 1327) from {username='slave'} at 127.0.0.1:49186
I20240503 02:14:38.215840  1873 tablet_replica.cc:330] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: stopping tablet replica
I20240503 02:14:38.216040  1873 raft_consensus.cc:2229] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:38.216200  1873 raft_consensus.cc:2258] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:38.217639  1873 ts_tablet_manager.cc:1875] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:38.224061  1873 ts_tablet_manager.cc:1888] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1326
I20240503 02:14:38.224193  1873 log.cc:1196] T 896ae927d34e46188120e705fb0ae361 P ecf926bd3fb84e0d8f66465d78a70d31: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/896ae927d34e46188120e705fb0ae361
I20240503 02:14:38.225453 27501 catalog_manager.cc:4830] TS ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): tablet 896ae927d34e46188120e705fb0ae361 (table rebalance_test_table_3 [id=c7a1858d2e11490bae75b4cee363974c]) successfully deleted
I20240503 02:14:38.389259   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:38.407454 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:38.417996 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:38.449637 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.459530 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:38.485204 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.512409 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:38.513185 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:38.619273 27915 consensus_queue.cc:237] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1248, Committed index: 1248, Last appended: 2.1248, Last appended by leader: 1247, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1249 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:38.621109   924 raft_consensus.cc:1261] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Refusing update from remote peer eaffa047cf9b43b79a5b75089ea63901: Log matching property violated. Preceding OpId in replica: term: 2 index: 1248. Preceding OpId from leader: term: 2 index: 1249. (index mismatch)
I20240503 02:14:38.622006  1840 consensus_queue.cc:1035] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1249, Last known committed idx: 1248, Time since last communication: 0.000s
I20240503 02:14:38.624905  1930 raft_consensus.cc:2932] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 [term 2 LEADER]: Committing config change with OpId 2.1249: config changed from index 1247 to 1249, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 1249 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.625736   924 raft_consensus.cc:2932] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Committing config change with OpId 2.1249: config changed from index 1247 to 1249, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New config: { opid_index: 1249 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.630319 27509 catalog_manager.cc:5484] T 4dd7e1c1b87f4f6db714735a13e25f67 P eaffa047cf9b43b79a5b75089ea63901 reported cstate change: config changed from index 1247 to 1249, VOTER acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "eaffa047cf9b43b79a5b75089ea63901" committed_config { opid_index: 1249 OBSOLETE_local: false peers { permanent_uuid: "eaffa047cf9b43b79a5b75089ea63901" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 45743 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } } }
I20240503 02:14:38.635996 27766 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dd7e1c1b87f4f6db714735a13e25f67 with delete_type TABLET_DATA_TOMBSTONED (TS acd475a9eee143a89c1d28a05fa51d26 not found in new config with opid_index 1249) from {username='slave'} at 127.0.0.1:51446
I20240503 02:14:38.636766  1936 tablet_replica.cc:330] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: stopping tablet replica
I20240503 02:14:38.636989  1936 raft_consensus.cc:2229] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:14:38.637208  1936 raft_consensus.cc:2258] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:14:38.639402  1936 ts_tablet_manager.cc:1875] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:14:38.646332  1936 ts_tablet_manager.cc:1888] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1248
I20240503 02:14:38.646522  1936 log.cc:1196] T 4dd7e1c1b87f4f6db714735a13e25f67 P acd475a9eee143a89c1d28a05fa51d26: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:38.647627 27499 catalog_manager.cc:4830] TS acd475a9eee143a89c1d28a05fa51d26 (127.20.218.66:46503): tablet 4dd7e1c1b87f4f6db714735a13e25f67 (table rebalance_test_table_2 [id=67a44eec1413407c807cc27bcea05dd2]) successfully deleted
I20240503 02:14:38.787375 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:38.801920 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.809044   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:38.820307 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:38.830614 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:38.854110 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:38.909478 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:38.949368 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:39.116047 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:39.134812 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:39.138682   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:39.145936 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:39.146183 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:39.156898 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:39.162858 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:39.179437 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:39.633850 29324 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:14:39.636415 29193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:39.647408 29063 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:14:39.660097   905 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:14:39.674779 27766 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:39.703472 27895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:14:39.756876 27635 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:14:39.760776 28932 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 e729927c47f74970b0a958b67ee29059 | 127.20.218.126:37537 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:39207 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 24868c13491843548e8da6583ef1e9d5 | 127.20.218.70:37393 | HEALTHY | <none>   |       0        |       0
 2ce83eb95475412c867e429ed43f0e65 | 127.20.218.72:39621 | HEALTHY | <none>   |       0        |       0
 94278d0a47dc47f2a0d5be9589f641be | 127.20.218.71:40065 | HEALTHY | <none>   |       2        |       0
 a8ef897d344843e2bbbbccc342f924ca | 127.20.218.69:32881 | HEALTHY | <none>   |       1        |       0
 acd475a9eee143a89c1d28a05fa51d26 | 127.20.218.66:46503 | HEALTHY | <none>   |       4        |       0
 b0eb49c52e794992a8fc34173f0cdcef | 127.20.218.68:39319 | HEALTHY | <none>   |       2        |       0
 eaffa047cf9b43b79a5b75089ea63901 | 127.20.218.67:45743 | HEALTHY | <none>   |       3        |       0
 ecf926bd3fb84e0d8f66465d78a70d31 | 127.20.218.65:44115 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.20.218.65                                                                                                                                                    | experimental          | 127.20.218.65:44115
 local_ip_for_outbound_sockets            | 127.20.218.66                                                                                                                                                    | experimental          | 127.20.218.66:46503
 local_ip_for_outbound_sockets            | 127.20.218.67                                                                                                                                                    | experimental          | 127.20.218.67:45743
 local_ip_for_outbound_sockets            | 127.20.218.68                                                                                                                                                    | experimental          | 127.20.218.68:39319
 local_ip_for_outbound_sockets            | 127.20.218.69                                                                                                                                                    | experimental          | 127.20.218.69:32881
 local_ip_for_outbound_sockets            | 127.20.218.70                                                                                                                                                    | experimental          | 127.20.218.70:37393
 local_ip_for_outbound_sockets            | 127.20.218.71                                                                                                                                                    | experimental          | 127.20.218.71:40065
 local_ip_for_outbound_sockets            | 127.20.218.72                                                                                                                                                    | experimental          | 127.20.218.72:39621
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.20.218.65:44115
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.20.218.66:46503
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.20.218.67:45743
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.20.218.68:39319
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.20.218.69:32881
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.20.218.70:37393
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.20.218.71:40065
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.20.218.72:39621

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:39207 | all 8 server(s) checked
 time_source         | builtin             | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:14:39.823315 21353 log_verifier.cc:126] Checking tablet 1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:14:40.277786 21353 log_verifier.cc:177] Verified matching terms for 1276 ops in tablet 1cc58692b4d144ed8bd4ad33cc8d022c
I20240503 02:14:40.279340 21353 log_verifier.cc:126] Checking tablet 376511df6d8645268134e125791f6aca
I20240503 02:14:40.477748 27504 maintenance_manager.cc:419] P e729927c47f74970b0a958b67ee29059: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033386
I20240503 02:14:40.508402 27503 maintenance_manager.cc:643] P e729927c47f74970b0a958b67ee29059: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.030s	user 0.021s	sys 0.006s Metrics: {"bytes_written":12613,"cfile_init":1,"dirs.queue_time_us":458,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1121,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":25,"thread_start_us":557,"threads_started":2,"wal-append.queue_time_us":340}
I20240503 02:14:40.509712 27504 maintenance_manager.cc:419] P e729927c47f74970b0a958b67ee29059: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 3331 bytes on disk
I20240503 02:14:40.510519 27503 maintenance_manager.cc:643] P e729927c47f74970b0a958b67ee29059: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 02:14:40.745719 21353 log_verifier.cc:177] Verified matching terms for 1213 ops in tablet 376511df6d8645268134e125791f6aca
I20240503 02:14:40.747200 21353 log_verifier.cc:126] Checking tablet 42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:14:40.759464 21353 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 42ad89cd14fb4dcfbfa7caeb944549c2
I20240503 02:14:40.759574 21353 log_verifier.cc:126] Checking tablet 4395a743162549ca96757c5838c0e190
I20240503 02:14:41.156476 21353 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 4395a743162549ca96757c5838c0e190
I20240503 02:14:41.157940 21353 log_verifier.cc:126] Checking tablet 4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:41.588222 21353 log_verifier.cc:177] Verified matching terms for 1249 ops in tablet 4dd7e1c1b87f4f6db714735a13e25f67
I20240503 02:14:41.589962 21353 log_verifier.cc:126] Checking tablet 4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:14:41.948015 21353 log_verifier.cc:177] Verified matching terms for 1134 ops in tablet 4ec65a7b8d0943119b929ae3a9544ec0
I20240503 02:14:41.949465 21353 log_verifier.cc:126] Checking tablet 6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:42.335271 21353 log_verifier.cc:177] Verified matching terms for 1128 ops in tablet 6af88acfe24e49279a98b5dcf417bed7
I20240503 02:14:42.336596 21353 log_verifier.cc:126] Checking tablet 6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:14:42.712891 21353 log_verifier.cc:177] Verified matching terms for 1192 ops in tablet 6b842b202e3b4d17afd245f0d90a34a4
I20240503 02:14:42.714177 21353 log_verifier.cc:126] Checking tablet 878e847516c74149966db83f6f70c6d0
I20240503 02:14:43.107654 21353 log_verifier.cc:177] Verified matching terms for 1147 ops in tablet 878e847516c74149966db83f6f70c6d0
I20240503 02:14:43.108839 21353 log_verifier.cc:126] Checking tablet 896ae927d34e46188120e705fb0ae361
I20240503 02:14:43.558679 21353 log_verifier.cc:177] Verified matching terms for 1327 ops in tablet 896ae927d34e46188120e705fb0ae361
I20240503 02:14:43.560400 21353 log_verifier.cc:126] Checking tablet 8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:14:43.949784 21353 log_verifier.cc:177] Verified matching terms for 1263 ops in tablet 8bf2a0fe803b42e4a706f50762a4b87d
I20240503 02:14:43.951267 21353 log_verifier.cc:126] Checking tablet 9698b9fc77ea4172b58f410383f2d926
I20240503 02:14:44.343062 21353 log_verifier.cc:177] Verified matching terms for 1162 ops in tablet 9698b9fc77ea4172b58f410383f2d926
I20240503 02:14:44.344393 21353 log_verifier.cc:126] Checking tablet a52a6040dd89493da32af4b283774f03
I20240503 02:14:44.715010 21353 log_verifier.cc:177] Verified matching terms for 1181 ops in tablet a52a6040dd89493da32af4b283774f03
I20240503 02:14:44.716256 21353 log_verifier.cc:126] Checking tablet a63be7e6f2da4163baced11894065444
I20240503 02:14:45.145200 21353 log_verifier.cc:177] Verified matching terms for 1232 ops in tablet a63be7e6f2da4163baced11894065444
I20240503 02:14:45.146848 21353 log_verifier.cc:126] Checking tablet ddd409a1c4554665831eceef070ab660
I20240503 02:14:45.527451 21353 log_verifier.cc:177] Verified matching terms for 1213 ops in tablet ddd409a1c4554665831eceef070ab660
I20240503 02:14:45.528908 21353 log_verifier.cc:126] Checking tablet f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:14:45.941244 21353 log_verifier.cc:177] Verified matching terms for 1154 ops in tablet f475da5fc141450ba4ddfe739ea3ad96
I20240503 02:14:45.951010 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27574
I20240503 02:14:45.969506 27696 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:46.298552 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27574
W20240503 02:14:46.357302 27851 connection.cc:535] server connection from 127.20.218.65:44729 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:46.357827 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27705
I20240503 02:14:46.364570 27826 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:46.681212 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27705
W20240503 02:14:46.741643 29019 connection.cc:535] server connection from 127.20.218.66:50083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:14:46.741647 28887 connection.cc:535] server connection from 127.20.218.66:33691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:46.742602 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27835
I20240503 02:14:46.748750 27957 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:47.057602 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27835
I20240503 02:14:47.120502 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 28870
I20240503 02:14:47.127787 28991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:47.138410  2065 raft_consensus.cc:482] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecf926bd3fb84e0d8f66465d78a70d31)
I20240503 02:14:47.138631  2065 raft_consensus.cc:504] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "a8ef897d344843e2bbbbccc342f924ca" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 32881 } }
I20240503 02:14:47.139221  2065 leader_election.cc:288] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
W20240503 02:14:47.140270 29018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
W20240503 02:14:47.141788 29018 leader_election.cc:334] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
I20240503 02:14:47.141912 29018 leader_election.cc:302] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: a8ef897d344843e2bbbbccc342f924ca; no voters: ecf926bd3fb84e0d8f66465d78a70d31
I20240503 02:14:47.142241  2065 raft_consensus.cc:2726] T 6b842b202e3b4d17afd245f0d90a34a4 P a8ef897d344843e2bbbbccc342f924ca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:47.333665  2074 raft_consensus.cc:482] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecf926bd3fb84e0d8f66465d78a70d31)
I20240503 02:14:47.333928  2074 raft_consensus.cc:504] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 656 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "94278d0a47dc47f2a0d5be9589f641be" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 40065 } }
I20240503 02:14:47.334681  2074 leader_election.cc:288] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
W20240503 02:14:47.336169 29279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
W20240503 02:14:47.337834 29279 leader_election.cc:334] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
I20240503 02:14:47.337985 29279 leader_election.cc:302] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 94278d0a47dc47f2a0d5be9589f641be; no voters: ecf926bd3fb84e0d8f66465d78a70d31
I20240503 02:14:47.338243  2074 raft_consensus.cc:2726] T 376511df6d8645268134e125791f6aca P 94278d0a47dc47f2a0d5be9589f641be [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:47.391110 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 28870
I20240503 02:14:47.444516 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 29001
I20240503 02:14:47.452407 29122 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:47.728412 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 29001
I20240503 02:14:47.785629 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 29131
I20240503 02:14:47.792285 29252 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:47.799278  2089 raft_consensus.cc:482] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecf926bd3fb84e0d8f66465d78a70d31)
I20240503 02:14:47.799610  2089 raft_consensus.cc:504] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 1128 OBSOLETE_local: false peers { permanent_uuid: "ecf926bd3fb84e0d8f66465d78a70d31" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 44115 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:47.801649  2089 leader_election.cc:288] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115)
W20240503 02:14:47.802829   860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
W20240503 02:14:47.804396   860 leader_election.cc:334] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ecf926bd3fb84e0d8f66465d78a70d31 (127.20.218.65:44115): Network error: Client connection negotiation failed: client connection to 127.20.218.65:44115: connect: Connection refused (error 111)
I20240503 02:14:47.804579   860 leader_election.cc:302] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2ce83eb95475412c867e429ed43f0e65; no voters: ecf926bd3fb84e0d8f66465d78a70d31
I20240503 02:14:47.804926  2089 raft_consensus.cc:2726] T 6af88acfe24e49279a98b5dcf417bed7 P 2ce83eb95475412c867e429ed43f0e65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:14:48.066532 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 29131
I20240503 02:14:48.117211 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 29261
I20240503 02:14:48.124900 29383 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:48.400662 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 29261
I20240503 02:14:48.426543  2096 raft_consensus.cc:482] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b0eb49c52e794992a8fc34173f0cdcef)
I20240503 02:14:48.426740  2096 raft_consensus.cc:504] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 1178 OBSOLETE_local: false peers { permanent_uuid: "b0eb49c52e794992a8fc34173f0cdcef" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 39319 } } peers { permanent_uuid: "2ce83eb95475412c867e429ed43f0e65" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 39621 } }
I20240503 02:14:48.427438  2096 leader_election.cc:288] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319)
W20240503 02:14:48.429662   860 leader_election.cc:334] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b0eb49c52e794992a8fc34173f0cdcef (127.20.218.68:39319): Network error: Client connection negotiation failed: client connection to 127.20.218.68:39319: connect: Connection refused (error 111)
I20240503 02:14:48.429850   860 leader_election.cc:302] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2ce83eb95475412c867e429ed43f0e65; no voters: b0eb49c52e794992a8fc34173f0cdcef
I20240503 02:14:48.430155  2096 raft_consensus.cc:2726] T 4395a743162549ca96757c5838c0e190 P 2ce83eb95475412c867e429ed43f0e65 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 02:14:48.453248 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 843
I20240503 02:14:48.460265   964 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:48.665532  2096 raft_consensus.cc:482] T 4dd7e1c1b87f4f6db714735a13e25f67 P 2ce83eb95475412c867e429ed43f0e65 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader eaffa047cf9b43b79a5b75089ea63901)
I20240503 02:14:48.665817 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 843
W20240503 02:14:48.701649 27499 connection.cc:535] server connection from 127.20.218.72:35289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:14:48.702066 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 27484
I20240503 02:14:48.703379 27545 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:48.964648 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 27484
2024-05-03T02:14:49Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/2 (129322 ms)
[----------] 1 test from RebalanceParamTest (129322 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240503 02:14:49.089309 21353 test_util.cc:268] Using random seed: 1220670719
I20240503 02:14:49.091275 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:14:49.091356 21353 ts_itest-base.cc:113] --------------
I20240503 02:14:49.091403 21353 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:14:49.091457 21353 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:14:49.091653 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:14:49Z Disabled control of system clock
I20240503 02:14:49.152526 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:35007
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:35007
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:14:49.535753  2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:49.536118  2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:49.536252  2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:49.544119  2109 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:49.544275  2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:14:49.544327  2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:49.544420  2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:14:49.544493  2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:14:49.554137  2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:35007
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:49.555608  2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:49.557574  2109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:49.569253  2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:49.569202  2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:49.569176  2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:49.570113  2109 server_base.cc:1034] running on GCE node
I20240503 02:14:49.571036  2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:49.572702  2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:49.573968  2109 hybrid_clock.cc:648] HybridClock initialized: now 1714702489573911 us; error 57 us; skew 500 ppm
I20240503 02:14:49.574385  2109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:49.577080  2109 webserver.cc:460] Webserver started at http://127.20.218.126:37743/ using document root <none> and password file <none>
I20240503 02:14:49.577973  2109 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:49.578146  2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:49.578502  2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:49.581465  2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6fad05a27a164335b985e3009b73dd44"
format_stamp: "Formatted at 2024-05-03 02:14:49 on dist-test-slave-81lq"
I20240503 02:14:49.582391  2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6fad05a27a164335b985e3009b73dd44"
format_stamp: "Formatted at 2024-05-03 02:14:49 on dist-test-slave-81lq"
I20240503 02:14:49.588289  2109 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:14:49.592618  2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:49.594314  2109 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:14:49.594532  2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6fad05a27a164335b985e3009b73dd44"
format_stamp: "Formatted at 2024-05-03 02:14:49 on dist-test-slave-81lq"
I20240503 02:14:49.594761  2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:49.614378  2109 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:49.615293  2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:49.615594  2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:49.635994  2109 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:35007
I20240503 02:14:49.636004  2174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:35007 every 8 connection(s)
I20240503 02:14:49.637535  2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:14:49.639794 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2109
I20240503 02:14:49.640018 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:14:49.642284  2175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:49.656392  2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44: Bootstrap starting.
I20240503 02:14:49.659612  2175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:49.660706  2175 log.cc:824] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:49.663475  2175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44: No bootstrap required, opened a new log
I20240503 02:14:49.669807  2175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } }
I20240503 02:14:49.670228  2175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:49.670336  2175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fad05a27a164335b985e3009b73dd44, State: Initialized, Role: FOLLOWER
I20240503 02:14:49.670958  2175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } }
I20240503 02:14:49.671182  2175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:14:49.671310  2175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:14:49.671471  2175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:49.673699  2175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } }
I20240503 02:14:49.674266  2175 leader_election.cc:302] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fad05a27a164335b985e3009b73dd44; no voters: 
I20240503 02:14:49.674927  2175 leader_election.cc:288] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:14:49.675073  2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:49.675778  2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [term 1 LEADER]: Becoming Leader. State: Replica: 6fad05a27a164335b985e3009b73dd44, State: Running, Role: LEADER
I20240503 02:14:49.676375  2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } }
I20240503 02:14:49.677078  2175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:14:49.679601  2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fad05a27a164335b985e3009b73dd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } } }
I20240503 02:14:49.679487  2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fad05a27a164335b985e3009b73dd44. Latest consensus state: current_term: 1 leader_uuid: "6fad05a27a164335b985e3009b73dd44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fad05a27a164335b985e3009b73dd44" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 35007 } } }
I20240503 02:14:49.679935  2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:49.679970  2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44 [sys.catalog]: This master's current role is: LEADER
I20240503 02:14:49.682528  2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:14:49.688510  2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:14:49.698376  2188 catalog_manager.cc:1347] Generated new cluster ID: f5be7ca8e4d04634bd6705e02e457aec
I20240503 02:14:49.698526  2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:14:49.710072  2188 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:14:49.711973  2188 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:14:49.722744  2188 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6fad05a27a164335b985e3009b73dd44: Generated new TSK 0
I20240503 02:14:49.723903  2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:14:49.738224 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:14:50.121740  2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:50.122072  2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:50.122218  2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:50.130555  2199 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:50.130712  2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:50.130887  2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:14:50.140300  2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:50.141943  2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:50.143865  2199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:50.156874  2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:50.157090  2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:50.157497  2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:50.157922  2199 server_base.cc:1034] running on GCE node
I20240503 02:14:50.158735  2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:50.160077  2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:50.161350  2199 hybrid_clock.cc:648] HybridClock initialized: now 1714702490161260 us; error 58 us; skew 500 ppm
I20240503 02:14:50.161823  2199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:50.164467  2199 webserver.cc:460] Webserver started at http://127.20.218.65:38855/ using document root <none> and password file <none>
I20240503 02:14:50.165300  2199 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:50.165498  2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:50.165854  2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:50.168574  2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "08055479ea104aa290d19955d3720850"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.169436  2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "08055479ea104aa290d19955d3720850"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.175243  2199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:14:50.179653  2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.181399  2199 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:14:50.181605  2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "08055479ea104aa290d19955d3720850"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.181813  2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:50.214987  2199 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:50.215981  2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:50.216300  2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:50.217443  2199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:50.219280  2199 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:50.219395  2199 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.219489  2199 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:50.219566  2199 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.256590  2199 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:34489
I20240503 02:14:50.256680  2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:34489 every 8 connection(s)
I20240503 02:14:50.258078  2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:14:50.263656 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2199
I20240503 02:14:50.263886 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:14:50.268054 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:50.272600  2326 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:50.273002  2326 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:50.273864  2326 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:50.276156  2140 ts_manager.cc:194] Registered new tserver with Master: 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:14:50.277956  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:35201
W20240503 02:14:50.640223  2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:50.640525  2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:50.640636  2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:50.648639  2330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:50.648790  2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:50.648977  2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:14:50.658885  2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:50.660274  2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:50.662132  2330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:50.675483  2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:50.675529  2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:50.675585  2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:50.676487  2330 server_base.cc:1034] running on GCE node
I20240503 02:14:50.677176  2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:50.678411  2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:50.679971  2330 hybrid_clock.cc:648] HybridClock initialized: now 1714702490679901 us; error 89 us; skew 500 ppm
I20240503 02:14:50.680359  2330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:50.683223  2330 webserver.cc:460] Webserver started at http://127.20.218.66:46751/ using document root <none> and password file <none>
I20240503 02:14:50.684031  2330 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:50.684144  2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:50.684482  2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:50.687583  2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6d54feb0bbc14c109783371d2939b234"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.688424  2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6d54feb0bbc14c109783371d2939b234"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.694389  2330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.004s
I20240503 02:14:50.698192  2344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.700120  2330 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:14:50.700353  2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6d54feb0bbc14c109783371d2939b234"
format_stamp: "Formatted at 2024-05-03 02:14:50 on dist-test-slave-81lq"
I20240503 02:14:50.700596  2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:50.715492  2330 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:50.716456  2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:50.716759  2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:50.717926  2330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:50.719609  2330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:50.719718  2330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.719838  2330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:50.719921  2330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:50.754781  2330 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:34903
I20240503 02:14:50.754828  2456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:34903 every 8 connection(s)
I20240503 02:14:50.756134  2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:14:50.757503 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2330
I20240503 02:14:50.757712 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:14:50.761837 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:50.773918  2457 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:50.774330  2457 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:50.775393  2457 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:50.777671  2140 ts_manager.cc:194] Registered new tserver with Master: 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:14:50.778918  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:57141
W20240503 02:14:51.136449  2460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:51.136790  2460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:51.136886  2460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:51.144732  2460 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:51.144886  2460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:51.145011  2460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:14:51.154564  2460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:51.155923  2460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:51.157755  2460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:51.170746  2465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:51.170998  2468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:51.171041  2466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:51.171761  2460 server_base.cc:1034] running on GCE node
I20240503 02:14:51.172420  2460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:51.173511  2460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:51.174773  2460 hybrid_clock.cc:648] HybridClock initialized: now 1714702491174714 us; error 39 us; skew 500 ppm
I20240503 02:14:51.175161  2460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:51.182245  2460 webserver.cc:460] Webserver started at http://127.20.218.67:39797/ using document root <none> and password file <none>
I20240503 02:14:51.183144  2460 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:51.183264  2460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:51.183575  2460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:51.186463  2460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "08bef4c144bc4e4d8cb5836aee8da852"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.187355  2460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "08bef4c144bc4e4d8cb5836aee8da852"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.193460  2460 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.006s
I20240503 02:14:51.197867  2474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.199688  2460 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
I20240503 02:14:51.199890  2460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "08bef4c144bc4e4d8cb5836aee8da852"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.200170  2460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:51.237149  2460 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:51.238134  2460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:51.238448  2460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:51.239537  2460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:51.241189  2460 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:51.241286  2460 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.241441  2460 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:51.241509  2460 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.275959  2460 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:38127
I20240503 02:14:51.276019  2586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:38127 every 8 connection(s)
I20240503 02:14:51.277434  2460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:14:51.282029  2326 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:51.284843 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2460
I20240503 02:14:51.285053 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:14:51.288834 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:51.296536  2587 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:51.296941  2587 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:51.298329  2587 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:51.300550  2140 ts_manager.cc:194] Registered new tserver with Master: 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:14:51.301635  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:41987
W20240503 02:14:51.676983  2591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:51.677409  2591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:51.677564  2591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:51.685667  2591 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:51.685822  2591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:51.685945  2591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:14:51.696013  2591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:51.697513  2591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:51.699595  2591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:51.713212  2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:51.713315  2596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:51.713250  2597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:51.713888  2591 server_base.cc:1034] running on GCE node
I20240503 02:14:51.714629  2591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:51.715754  2591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:51.716996  2591 hybrid_clock.cc:648] HybridClock initialized: now 1714702491716936 us; error 53 us; skew 500 ppm
I20240503 02:14:51.717393  2591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:51.720074  2591 webserver.cc:460] Webserver started at http://127.20.218.68:42393/ using document root <none> and password file <none>
I20240503 02:14:51.721297  2591 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:51.721536  2591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:51.721974  2591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:51.725569  2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e9fdaf326bb64edabc43b366df870f83"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.726527  2591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e9fdaf326bb64edabc43b366df870f83"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.732450  2591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.003s
I20240503 02:14:51.736596  2605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.738852  2591 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:14:51.739109  2591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e9fdaf326bb64edabc43b366df870f83"
format_stamp: "Formatted at 2024-05-03 02:14:51 on dist-test-slave-81lq"
I20240503 02:14:51.739363  2591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:51.776614  2591 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:51.777734  2591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:51.778074  2591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:51.779213  2591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:51.781020  2591 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:51.781141  2591 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.781263  2591 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:51.781358  2591 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:51.782615  2457 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:51.820592  2591 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:38103
I20240503 02:14:51.820639  2717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:38103 every 8 connection(s)
I20240503 02:14:51.822016  2591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:14:51.831058 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2591
I20240503 02:14:51.831292 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:14:51.835166 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:0
--local_ip_for_outbound_sockets=127.20.218.69
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:51.838317  2718 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:51.838702  2718 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:51.839587  2718 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:51.841257  2140 ts_manager.cc:194] Registered new tserver with Master: e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:14:51.842487  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:58961
W20240503 02:14:52.216003  2722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:52.216385  2722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:52.216542  2722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:52.224668  2722 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:52.224869  2722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:52.225085  2722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:14:52.239035  2722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:52.240944  2722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:52.243495  2722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:52.258543  2730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:52.258453  2727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:52.259466  2728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:52.260975  2722 server_base.cc:1034] running on GCE node
I20240503 02:14:52.261791  2722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:52.262867  2722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:52.264103  2722 hybrid_clock.cc:648] HybridClock initialized: now 1714702492264035 us; error 63 us; skew 500 ppm
I20240503 02:14:52.264490  2722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:52.267201  2722 webserver.cc:460] Webserver started at http://127.20.218.69:35331/ using document root <none> and password file <none>
I20240503 02:14:52.268047  2722 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:52.268162  2722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:52.268529  2722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:52.271458  2722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1d36807868fe4a82bca42c25831ce8a2"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.272365  2722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1d36807868fe4a82bca42c25831ce8a2"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.279245  2722 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.001s
I20240503 02:14:52.284528  2736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.286306  2722 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:14:52.286490  2722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1d36807868fe4a82bca42c25831ce8a2"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.286914  2722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:52.304327  2722 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:52.304986  2587 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:52.305594  2722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:52.305928  2722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:52.307214  2722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:52.308864  2722 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:52.309005  2722 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.309137  2722 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:52.309224  2722 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.352555  2722 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:39741
I20240503 02:14:52.352594  2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:39741 every 8 connection(s)
I20240503 02:14:52.354103  2722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:14:52.358168 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2722
I20240503 02:14:52.358392 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:14:52.363082 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:0
--local_ip_for_outbound_sockets=127.20.218.70
--webserver_interface=127.20.218.70
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:52.371908  2849 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:52.372262  2849 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:52.373189  2849 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:52.374684  2140 ts_manager.cc:194] Registered new tserver with Master: 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:14:52.375500  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:33903
W20240503 02:14:52.752478  2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:52.752820  2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:52.752934  2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:52.761179  2853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:52.761405  2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:52.761615  2853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:14:52.771615  2853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:52.773082  2853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:52.775069  2853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:52.789556  2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:52.789710  2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:52.789677  2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:52.790761  2853 server_base.cc:1034] running on GCE node
I20240503 02:14:52.791494  2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:52.792717  2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:52.793983  2853 hybrid_clock.cc:648] HybridClock initialized: now 1714702492793919 us; error 81 us; skew 500 ppm
I20240503 02:14:52.794389  2853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:52.797214  2853 webserver.cc:460] Webserver started at http://127.20.218.70:40999/ using document root <none> and password file <none>
I20240503 02:14:52.798496  2853 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:52.798646  2853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:52.799057  2853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:52.802987  2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9e699d9f984b4cf2b8b468322759b78c"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.804016  2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9e699d9f984b4cf2b8b468322759b78c"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.811461  2853 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.001s	sys 0.007s
I20240503 02:14:52.816885  2867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.818578  2853 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.002s
I20240503 02:14:52.818794  2853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e699d9f984b4cf2b8b468322759b78c"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:14:52.819041  2853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:52.834522  2853 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:52.835505  2853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:52.835816  2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:52.836885  2853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:52.838582  2853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:52.838694  2853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.838842  2853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:52.838927  2853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:52.845817  2718 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:52.878190  2853 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:33893
I20240503 02:14:52.878234  2979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:33893 every 8 connection(s)
I20240503 02:14:52.879617  2853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:14:52.879951 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2853
I20240503 02:14:52.880127 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:14:52.884068 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:0
--local_ip_for_outbound_sockets=127.20.218.71
--webserver_interface=127.20.218.71
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:52.899776  2980 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:52.900126  2980 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:52.900957  2980 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:52.902545  2140 ts_manager.cc:194] Registered new tserver with Master: 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:14:52.903406  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:47201
W20240503 02:14:53.263496  2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:53.263844  2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:53.263998  2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:53.271929  2984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:53.272122  2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:53.272305  2984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:14:53.282688  2984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:53.284150  2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:53.286121  2984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:53.300256  2990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:53.300371  2989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:53.300372  2992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:53.301916  2984 server_base.cc:1034] running on GCE node
I20240503 02:14:53.302655  2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:53.303758  2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:53.304984  2984 hybrid_clock.cc:648] HybridClock initialized: now 1714702493304937 us; error 40 us; skew 500 ppm
I20240503 02:14:53.305507  2984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:53.308531  2984 webserver.cc:460] Webserver started at http://127.20.218.71:42735/ using document root <none> and password file <none>
I20240503 02:14:53.309700  2984 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:53.309865  2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:53.310302  2984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:53.313755  2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "408af0ab5520465e942cc714bfc6c73b"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.314654  2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "408af0ab5520465e942cc714bfc6c73b"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.320546  2984 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.001s	sys 0.004s
I20240503 02:14:53.324931  2998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.327191  2984 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:14:53.327489  2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "408af0ab5520465e942cc714bfc6c73b"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.327760  2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:53.358369  2984 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:53.359468  2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:53.359738  2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:53.360885  2984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:53.362600  2984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:53.362708  2984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.362819  2984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:53.362879  2984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.378346  2849 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:53.400348  2984 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:38285
I20240503 02:14:53.400411  3110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:38285 every 8 connection(s)
I20240503 02:14:53.401916  2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:14:53.406054 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 2984
I20240503 02:14:53.406272 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:14:53.409711 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.72:0
--local_ip_for_outbound_sockets=127.20.218.72
--webserver_interface=127.20.218.72
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:14:53.416648  3111 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:53.417028  3111 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:53.417937  3111 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:53.419626  2140 ts_manager.cc:194] Registered new tserver with Master: 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:14:53.420460  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:38179
W20240503 02:14:53.789911  3115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:53.790311  3115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:14:53.790416  3115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:14:53.798595  3115 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:14:53.798766  3115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:14:53.798888  3115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.72
I20240503 02:14:53.809063  3115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.218.72
--webserver_port=0
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.72
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:14:53.810530  3115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:14:53.812669  3115 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:14:53.829020  3121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:53.829175  3123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:14:53.829057  3120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:14:53.830276  3115 server_base.cc:1034] running on GCE node
I20240503 02:14:53.831168  3115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:14:53.832556  3115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:14:53.833915  3115 hybrid_clock.cc:648] HybridClock initialized: now 1714702493833847 us; error 64 us; skew 500 ppm
I20240503 02:14:53.834452  3115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:14:53.837548  3115 webserver.cc:460] Webserver started at http://127.20.218.72:42969/ using document root <none> and password file <none>
I20240503 02:14:53.838441  3115 fs_manager.cc:359] Metadata directory not provided
I20240503 02:14:53.838575  3115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:14:53.838974  3115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:14:53.842532  3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7fd07044862a481aa387698017467c01"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.843636  3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7fd07044862a481aa387698017467c01"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.850939  3115 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.007s	sys 0.000s
I20240503 02:14:53.856122  3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.858207  3115 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:14:53.858438  3115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7fd07044862a481aa387698017467c01"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:14:53.858707  3115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:14:53.884691  3115 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:14:53.885769  3115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:14:53.886065  3115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:14:53.887357  3115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:14:53.889039  3115 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:14:53.889139  3115 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.889266  3115 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:14:53.889360  3115 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:14:53.907095  2980 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:14:53.928299  3115 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.72:37033
I20240503 02:14:53.928367  3241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.72:37033 every 8 connection(s)
I20240503 02:14:53.929870  3115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:14:53.938341 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 3115
I20240503 02:14:53.938560 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:14:53.946106  3242 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:14:53.946439  3242 heartbeater.cc:464] Registering TS with master...
I20240503 02:14:53.947281  3242 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:14:53.948944  2140 ts_manager.cc:194] Registered new tserver with Master: 7fd07044862a481aa387698017467c01 (127.20.218.72:37033)
I20240503 02:14:53.949869  2140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.72:42361
I20240503 02:14:53.956720 21353 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:14:53.977095 21353 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:14:53.977286 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2722
I20240503 02:14:53.987471  2844 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:54.074784 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2722
I20240503 02:14:54.095417 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2853
I20240503 02:14:54.106475  2975 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:54.190042 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2853
I20240503 02:14:54.208499 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2984
I20240503 02:14:54.218320  3106 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:54.304931 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2984
I20240503 02:14:54.323784 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 3115
I20240503 02:14:54.334206  3237 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:14:54.417837 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 3115
I20240503 02:14:58.187883  2140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:14:58.252538  2392 tablet_service.cc:1467] Processing CreateTablet for tablet 8870feaf47a64a929dd2e584ece47218 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.254309  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 734027e195414b15811ac9d32b132db8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.255522  2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.257443  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 734027e195414b15811ac9d32b132db8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.257465  2522 tablet_service.cc:1467] Processing CreateTablet for tablet 8870feaf47a64a929dd2e584ece47218 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.258806  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734027e195414b15811ac9d32b132db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.259044  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.259052  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734027e195414b15811ac9d32b132db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.261659  2259 tablet_service.cc:1467] Processing CreateTablet for tablet 190ebb1335d64b279a41fa38feff22b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.262266  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190ebb1335d64b279a41fa38feff22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.266332  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 190ebb1335d64b279a41fa38feff22b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.266980  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190ebb1335d64b279a41fa38feff22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.267230  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 8870feaf47a64a929dd2e584ece47218 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.267704  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.272045  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 734027e195414b15811ac9d32b132db8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.273787  2652 tablet_service.cc:1467] Processing CreateTablet for tablet 190ebb1335d64b279a41fa38feff22b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:58.277616  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190ebb1335d64b279a41fa38feff22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.281737  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734027e195414b15811ac9d32b132db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:58.283223  3272 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:14:58.288399  3273 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:14:58.289481  3272 tablet_bootstrap.cc:654] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.291240  3272 log.cc:824] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:58.296309  3273 tablet_bootstrap.cc:654] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.298763  3273 log.cc:824] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:58.301985  3272 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:14:58.302713  3272 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.013s
I20240503 02:14:58.316272  3273 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:14:58.316704  3273 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.011s
I20240503 02:14:58.316637  3272 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.317183  3272 raft_consensus.cc:374] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.317308  3272 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.318145  3272 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.318982  3274 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:14:58.327558  3274 tablet_bootstrap.cc:654] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.329779  3274 log.cc:824] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:58.331501  3278 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:14:58.332720  3273 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.333142  3272 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.030s	user 0.013s	sys 0.016s
I20240503 02:14:58.333643  3272 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:14:58.335633  3272 tablet_bootstrap.cc:654] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.335775  3278 tablet_bootstrap.cc:654] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.336889  3278 log.cc:824] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Log is configured to *not* fsync() on all Append() calls
I20240503 02:14:58.333266  3273 raft_consensus.cc:374] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.340986  3273 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.342706  3274 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:14:58.342797  3273 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.343377  3274 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.025s	user 0.005s	sys 0.008s
I20240503 02:14:58.346261  3272 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:14:58.346457  3272 ts_tablet_manager.cc:1367] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20240503 02:14:58.346745  3273 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.030s	user 0.012s	sys 0.007s
I20240503 02:14:58.347210  3273 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:14:58.347584  3272 raft_consensus.cc:348] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.347810  3272 raft_consensus.cc:374] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.347904  3272 raft_consensus.cc:729] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.348155  3272 consensus_queue.cc:260] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.348706  3272 ts_tablet_manager.cc:1398] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:14:58.349030  3273 tablet_bootstrap.cc:654] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.350142  3278 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:14:58.350612  3278 ts_tablet_manager.cc:1367] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.007s
I20240503 02:14:58.355681  3274 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.356141  3274 raft_consensus.cc:374] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.356261  3274 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.357125  3274 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.357754  3273 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:14:58.357954  3273 ts_tablet_manager.cc:1367] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240503 02:14:58.359105  3273 raft_consensus.cc:348] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.359360  3273 raft_consensus.cc:374] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.359474  3273 raft_consensus.cc:729] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.359731  3273 consensus_queue.cc:260] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.360383  3273 ts_tablet_manager.cc:1398] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:14:58.362360  3274 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.019s	user 0.011s	sys 0.003s
I20240503 02:14:58.362221  3278 raft_consensus.cc:348] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.362851  3278 raft_consensus.cc:374] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.362973  3278 raft_consensus.cc:729] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.363094  3274 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:14:58.363862  3278 consensus_queue.cc:260] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.365500  3274 tablet_bootstrap.cc:654] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.366215  3278 ts_tablet_manager.cc:1398] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.015s	user 0.011s	sys 0.004s
I20240503 02:14:58.366875  3278 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:14:58.368850  3278 tablet_bootstrap.cc:654] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.369728  3274 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:14:58.369963  3274 ts_tablet_manager.cc:1367] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.002s
I20240503 02:14:58.371016  3274 raft_consensus.cc:348] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.371209  3274 raft_consensus.cc:374] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.371315  3274 raft_consensus.cc:729] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.371488  3278 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:14:58.371568  3274 consensus_queue.cc:260] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.371676  3278 ts_tablet_manager.cc:1367] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:14:58.372561  3278 raft_consensus.cc:348] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.372784  3278 raft_consensus.cc:374] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.372893  3278 raft_consensus.cc:729] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.372869  3274 ts_tablet_manager.cc:1398] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:14:58.373167  3274 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:14:58.373085  3278 consensus_queue.cc:260] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.373682  3278 ts_tablet_manager.cc:1398] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:14:58.374871  3274 tablet_bootstrap.cc:654] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:58.377419  3274 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:14:58.377650  3274 ts_tablet_manager.cc:1367] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:14:58.378305  3274 raft_consensus.cc:348] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.378497  3274 raft_consensus.cc:374] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:58.378587  3274 raft_consensus.cc:729] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:14:58.378816  3274 consensus_queue.cc:260] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.379325  3274 ts_tablet_manager.cc:1398] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:14:58.463184  3284 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:58.463467  3284 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.465281  3284 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:14:58.472963  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:14:58.473007  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:14:58.473520  2673 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:14:58.473528  2412 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:14:58.474210  2216 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:14:58.474648  3284 raft_consensus.cc:2781] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:58.474819  3284 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:58.474941  3284 raft_consensus.cc:3037] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.477290  3284 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.478060  3284 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:14:58.478731  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:14:58.478718  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:14:58.478968  2412 raft_consensus.cc:3037] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.478971  2673 raft_consensus.cc:3037] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.481503  2673 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:14:58.481503  2412 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:14:58.482033  2214 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:58.482385  3284 raft_consensus.cc:2781] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:58.482906  3284 raft_consensus.cc:686] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:14:58.483394  3284 consensus_queue.cc:237] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.488399  2139 catalog_manager.cc:5484] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:14:58.504618  3277 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:58.504858  3277 raft_consensus.cc:504] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.506856  3277 leader_election.cc:288] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:14:58.512527  3285 raft_consensus.cc:482] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:58.512760  3285 raft_consensus.cc:504] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.515036  3285 leader_election.cc:288] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:14:58.514998  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:14:58.515103  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:14:58.515478  2281 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
W20240503 02:14:58.515549  2458 tablet.cc:2369] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:58.515638  2542 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:14:58.516559  2346 leader_election.cc:302] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:58.517073  3277 raft_consensus.cc:2781] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:58.517215  3277 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:58.517313  3277 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:14:58.517370  2327 tablet.cc:2369] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:58.519865  3277 raft_consensus.cc:504] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.520713  3277 leader_election.cc:288] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:14:58.521373  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:14:58.521370  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:14:58.521620  2281 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.521625  2542 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.524731  2281 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:14:58.524888  2542 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:14:58.525699  2346 leader_election.cc:302] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:58.526441  3277 raft_consensus.cc:2781] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:58.526711  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190ebb1335d64b279a41fa38feff22b4" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:14:58.526975  2542 raft_consensus.cc:2454] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:14:58.527098  3277 raft_consensus.cc:686] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:14:58.528043  2607 leader_election.cc:302] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:14:58.528136  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190ebb1335d64b279a41fa38feff22b4" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:14:58.528043  3277 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:58.528376  2281 raft_consensus.cc:2454] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:14:58.528506  3285 raft_consensus.cc:2781] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:58.528636  3285 raft_consensus.cc:482] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:58.528707  3285 raft_consensus.cc:3037] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.532009  3285 raft_consensus.cc:504] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.532373  2139 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:14:58.532987  3285 leader_election.cc:288] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:14:58.533519  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190ebb1335d64b279a41fa38feff22b4" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:14:58.533783  2281 raft_consensus.cc:3037] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:58.533821  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190ebb1335d64b279a41fa38feff22b4" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:14:58.534058  2542 raft_consensus.cc:3037] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:14:58.536167  2588 tablet.cc:2369] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:58.536720  2281 raft_consensus.cc:2454] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:14:58.537511  2609 leader_election.cc:302] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:14:58.537878  3285 raft_consensus.cc:2781] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:58.538457  2542 raft_consensus.cc:2454] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:14:58.538462  3285 raft_consensus.cc:686] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:14:58.538981  3285 consensus_queue.cc:237] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:14:58.543381  2139 catalog_manager.cc:5484] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
W20240503 02:14:58.579960  2719 tablet.cc:2369] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:14:58.974587  3285 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:58.981632  3301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:14:58.982214  3301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:14:58.982547  3299 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:14:58.996327  3301 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:14:59.003970  3277 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:59.016099  3297 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:14:59.075501  3284 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:14:59.092693  3335 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 299.682 ms
  time per row: 0.149841 ms
I20240503 02:14:59.463742  2135 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:14:59.475000  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 1eeb69e5eb3c459ca749aa37672ec057 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.475128  2392 tablet_service.cc:1467] Processing CreateTablet for tablet 1eeb69e5eb3c459ca749aa37672ec057 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.475524  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.475598  2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.476131  2652 tablet_service.cc:1467] Processing CreateTablet for tablet f084f1783d4d477e9c3791984e681f4c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.476578  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f1783d4d477e9c3791984e681f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.478063  2390 tablet_service.cc:1467] Processing CreateTablet for tablet 9253de3f798f4a208d566765ed2ba54d (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.478622  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253de3f798f4a208d566765ed2ba54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.481034  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 1eeb69e5eb3c459ca749aa37672ec057 (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.481130  2391 tablet_service.cc:1467] Processing CreateTablet for tablet f084f1783d4d477e9c3791984e681f4c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.481664  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.481921  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f1783d4d477e9c3791984e681f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.481842  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 9253de3f798f4a208d566765ed2ba54d (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.482249  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253de3f798f4a208d566765ed2ba54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.486459  2522 tablet_service.cc:1467] Processing CreateTablet for tablet f084f1783d4d477e9c3791984e681f4c (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.487109  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f1783d4d477e9c3791984e681f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.489076  3375 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:14:59.490160  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 9253de3f798f4a208d566765ed2ba54d (DEFAULT_TABLE table=rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:14:59.492478  3375 tablet_bootstrap.cc:654] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.493280  3376 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:14:59.494073  3374 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:14:59.495316  3376 tablet_bootstrap.cc:654] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.496914  3374 tablet_bootstrap.cc:654] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.504124  3376 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:14:59.504264  3375 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:14:59.504367  3376 ts_tablet_manager.cc:1367] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240503 02:14:59.504415  3375 ts_tablet_manager.cc:1367] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.004s
I20240503 02:14:59.505353  3375 raft_consensus.cc:348] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.505419  3376 raft_consensus.cc:348] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.505601  3375 raft_consensus.cc:374] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.505618  3376 raft_consensus.cc:374] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.505695  3375 raft_consensus.cc:729] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.505764  3376 raft_consensus.cc:729] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.505966  3375 consensus_queue.cc:260] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.506029  3376 consensus_queue.cc:260] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.506577  3375 ts_tablet_manager.cc:1398] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:14:59.506791  3376 ts_tablet_manager.cc:1398] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:14:59.506906  3375 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:14:59.507798  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253de3f798f4a208d566765ed2ba54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:14:59.509218  3375 tablet_bootstrap.cc:654] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.517716  3374 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:14:59.517978  3374 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.024s	user 0.005s	sys 0.013s
I20240503 02:14:59.519080  3374 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.519399  3374 raft_consensus.cc:374] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.519508  3374 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.519840  3374 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.523252  3374 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:14:59.523742  3374 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:14:59.528757  3374 tablet_bootstrap.cc:654] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.536245  3380 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:14:59.537719  3374 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:14:59.537925  3374 ts_tablet_manager.cc:1367] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20240503 02:14:59.538005  3375 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:14:59.538197  3375 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.016s
I20240503 02:14:59.538417  3380 tablet_bootstrap.cc:654] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.538771  3374 raft_consensus.cc:348] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.539028  3374 raft_consensus.cc:374] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.539117  3374 raft_consensus.cc:729] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.539058  3375 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.539300  3375 raft_consensus.cc:374] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.539391  3375 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.539398  3374 consensus_queue.cc:260] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.539685  3375 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.540104  3374 ts_tablet_manager.cc:1398] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:14:59.540442  3374 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:14:59.540243  3375 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:14:59.542186  3374 tablet_bootstrap.cc:654] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.544281  3374 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:14:59.544515  3374 ts_tablet_manager.cc:1367] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:14:59.545397  3380 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:14:59.545578  3380 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.004s
I20240503 02:14:59.545452  3374 raft_consensus.cc:348] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.545670  3374 raft_consensus.cc:374] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.545769  3374 raft_consensus.cc:729] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.546003  3374 consensus_queue.cc:260] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.546326  3380 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.546505  3374 ts_tablet_manager.cc:1398] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:14:59.546536  3380 raft_consensus.cc:374] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.546629  3380 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.546885  3380 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.547753  3380 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:14:59.548146  3380 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:14:59.549950  3380 tablet_bootstrap.cc:654] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.552363  3380 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:14:59.552525  3380 ts_tablet_manager.cc:1367] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:14:59.553406  3380 raft_consensus.cc:348] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.553617  3380 raft_consensus.cc:374] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.553676  3380 raft_consensus.cc:729] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.553947  3380 consensus_queue.cc:260] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.554411  3380 ts_tablet_manager.cc:1398] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:14:59.554852  3380 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:14:59.556385  3380 tablet_bootstrap.cc:654] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:14:59.558494  3380 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:14:59.558629  3380 ts_tablet_manager.cc:1367] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:14:59.559263  3380 raft_consensus.cc:348] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.559486  3380 raft_consensus.cc:374] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:14:59.559595  3380 raft_consensus.cc:729] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:14:59.559844  3380 consensus_queue.cc:260] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.560359  3380 ts_tablet_manager.cc:1398] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:14:59.663522  3277 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:59.663728  3277 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.664530  3277 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:14:59.665311  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:14:59.665632  2542 raft_consensus.cc:2454] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:14:59.666177  2346 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:59.666666  3277 raft_consensus.cc:2781] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:59.666842  3277 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:59.666975  3277 raft_consensus.cc:3037] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.670159  3277 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.671022  3277 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:14:59.671732  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:14:59.671967  2542 raft_consensus.cc:3037] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.673240  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:14:59.673563  2673 raft_consensus.cc:2454] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:14:59.673446  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:14:59.673731  2672 raft_consensus.cc:3037] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.674440  2542 raft_consensus.cc:2454] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:14:59.674990  2346 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:59.675467  3277 raft_consensus.cc:2781] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:59.675663  3277 raft_consensus.cc:686] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:14:59.676407  2672 raft_consensus.cc:2454] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:14:59.676256  3277 consensus_queue.cc:237] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.679468  2139 catalog_manager.cc:5484] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:14:59.693699  3385 raft_consensus.cc:482] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:59.693913  3385 raft_consensus.cc:504] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.695717  3385 leader_election.cc:288] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:14:59.715663  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253de3f798f4a208d566765ed2ba54d" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:14:59.715708  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253de3f798f4a208d566765ed2ba54d" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:14:59.715929  2281 raft_consensus.cc:2454] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:14:59.715940  2412 raft_consensus.cc:2454] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:14:59.716491  2478 leader_election.cc:302] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:14:59.716970  3385 raft_consensus.cc:2781] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:59.717095  3385 raft_consensus.cc:482] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:59.717190  3385 raft_consensus.cc:3037] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.719314  3385 raft_consensus.cc:504] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.720043  3385 leader_election.cc:288] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:14:59.720578  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253de3f798f4a208d566765ed2ba54d" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:14:59.720829  2412 raft_consensus.cc:3037] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.720724  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9253de3f798f4a208d566765ed2ba54d" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:14:59.720996  2281 raft_consensus.cc:3037] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.722980  2412 raft_consensus.cc:2454] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:14:59.723223  2281 raft_consensus.cc:2454] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:14:59.723477  2475 leader_election.cc:302] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:14:59.723829  3385 raft_consensus.cc:2781] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:59.724287  3385 raft_consensus.cc:686] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:14:59.724792  3385 consensus_queue.cc:237] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:14:59.729301  2139 catalog_manager.cc:5484] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:14:59.844081  3299 raft_consensus.cc:482] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:14:59.844291  3299 raft_consensus.cc:504] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.845034  3299 leader_election.cc:288] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:14:59.845877  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eeb69e5eb3c459ca749aa37672ec057" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:14:59.846176  2542 raft_consensus.cc:2454] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:14:59.846616  2607 leader_election.cc:302] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:14:59.846956  3299 raft_consensus.cc:2781] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:14:59.847047  3299 raft_consensus.cc:482] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:14:59.847163  3299 raft_consensus.cc:3037] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.849702  3299 raft_consensus.cc:504] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.850857  3299 leader_election.cc:288] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:14:59.851034  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eeb69e5eb3c459ca749aa37672ec057" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:14:59.851307  2542 raft_consensus.cc:3037] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:14:59.852612  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eeb69e5eb3c459ca749aa37672ec057" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:14:59.852741  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eeb69e5eb3c459ca749aa37672ec057" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:14:59.853029  2412 raft_consensus.cc:2454] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:14:59.853726  2542 raft_consensus.cc:2454] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:14:59.854272  2607 leader_election.cc:302] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 6d54feb0bbc14c109783371d2939b234
I20240503 02:14:59.854635  3299 raft_consensus.cc:2781] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:14:59.855108  3299 raft_consensus.cc:686] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:14:59.855410  3299 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:14:59.858731  2139 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:00.083459  3277 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:00.089676  3277 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:00.260277  3299 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:00.265600  2412 raft_consensus.cc:3037] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:00.269392  3397 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:00.269598  3399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:00.270042  3399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:00.280898  3399 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:00.282637  3385 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:00.304952  3385 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 238.95 ms
  time per row: 0.119475 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:15:01.651317  2140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:01.662178  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 0137e646f1bd452b950d870107afa060 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.662390  2652 tablet_service.cc:1467] Processing CreateTablet for tablet 0137e646f1bd452b950d870107afa060 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.662842  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0137e646f1bd452b950d870107afa060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.662840  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0137e646f1bd452b950d870107afa060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.662834  2521 tablet_service.cc:1467] Processing CreateTablet for tablet d2351807a2ed47419f8d674fbeb9d0ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.663357  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.663331  2653 tablet_service.cc:1467] Processing CreateTablet for tablet d2351807a2ed47419f8d674fbeb9d0ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.663786  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.666406  2260 tablet_service.cc:1467] Processing CreateTablet for tablet d2351807a2ed47419f8d674fbeb9d0ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.667068  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.670405  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 0137e646f1bd452b950d870107afa060 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.670925  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0137e646f1bd452b950d870107afa060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.675822  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 51224b6355ed41dbb0703b7feb73c223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.676599  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51224b6355ed41dbb0703b7feb73c223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.679206  3466 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:01.680322  3465 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:01.681066  3466 tablet_bootstrap.cc:654] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.683113  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 51224b6355ed41dbb0703b7feb73c223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.683789  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51224b6355ed41dbb0703b7feb73c223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.685215  2390 tablet_service.cc:1467] Processing CreateTablet for tablet 51224b6355ed41dbb0703b7feb73c223 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:01.685876  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51224b6355ed41dbb0703b7feb73c223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:01.682379  3465 tablet_bootstrap.cc:654] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.688468  3466 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:01.688658  3466 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240503 02:15:01.690251  3466 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.690515  3466 raft_consensus.cc:374] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.690694  3466 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.691023  3466 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.692405  3466 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:15:01.693015  3466 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:01.695775  3466 tablet_bootstrap.cc:654] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.726614  3470 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:01.730224  3470 tablet_bootstrap.cc:654] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.735798  3466 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:01.735817  3465 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:01.735999  3465 ts_tablet_manager.cc:1367] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.056s	user 0.017s	sys 0.031s
I20240503 02:15:01.736158  3466 ts_tablet_manager.cc:1367] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.043s	user 0.009s	sys 0.032s
I20240503 02:15:01.736955  3465 raft_consensus.cc:348] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.736975  3466 raft_consensus.cc:348] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.737238  3465 raft_consensus.cc:374] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.737293  3466 raft_consensus.cc:374] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.737401  3465 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.737728  3465 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.738011  3466 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.738380  3465 ts_tablet_manager.cc:1398] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:01.738548  3464 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:01.738402  3466 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.738809  3465 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:01.740644  3464 tablet_bootstrap.cc:654] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.742861  3466 ts_tablet_manager.cc:1398] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240503 02:15:01.743242  3466 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:01.743491  3465 tablet_bootstrap.cc:654] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.744953  3466 tablet_bootstrap.cc:654] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.748171  3470 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:01.748165  3465 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:01.748397  3465 ts_tablet_manager.cc:1367] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.000s
I20240503 02:15:01.748466  3470 ts_tablet_manager.cc:1367] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.004s
I20240503 02:15:01.749210  3464 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:01.749430  3464 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.000s
I20240503 02:15:01.749365  3465 raft_consensus.cc:348] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.749660  3465 raft_consensus.cc:374] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.749774  3465 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.749578  3470 raft_consensus.cc:348] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.749930  3470 raft_consensus.cc:374] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.750082  3470 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.750092  3465 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.750492  3470 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:01.750352  3464 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.750720  3465 ts_tablet_manager.cc:1398] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:01.750708  3464 raft_consensus.cc:374] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.750819  3464 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.751118  3465 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:01.751221  3470 ts_tablet_manager.cc:1398] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:15:01.751557  3470 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:01.751160  3464 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.753518  3470 tablet_bootstrap.cc:654] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.753647  3465 tablet_bootstrap.cc:654] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:01.753999  3464 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:01.756155  3466 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:01.756542  3466 ts_tablet_manager.cc:1367] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.005s
I20240503 02:15:01.759583  3466 raft_consensus.cc:348] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.760196  3466 raft_consensus.cc:374] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.760383  3466 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.760774  3466 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.762142  3466 ts_tablet_manager.cc:1398] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240503 02:15:01.769640  3465 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:01.769721  3470 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:01.769937  3470 ts_tablet_manager.cc:1367] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.018s	user 0.000s	sys 0.009s
I20240503 02:15:01.769960  3465 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.005s
I20240503 02:15:01.770782  3470 raft_consensus.cc:348] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.770797  3465 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.771016  3465 raft_consensus.cc:374] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.771050  3470 raft_consensus.cc:374] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:01.771135  3470 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.771158  3465 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:01.771395  3465 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.771430  3470 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.772024  3470 ts_tablet_manager.cc:1398] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:01.772121  3465 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:01.876103  3459 raft_consensus.cc:482] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:01.876327  3459 raft_consensus.cc:504] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.877126  3459 leader_election.cc:288] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:01.877749  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:01.877754  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:01.878017  2412 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:01.878047  2672 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:01.878506  2216 leader_election.cc:302] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:01.878815  3459 raft_consensus.cc:2781] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:01.878927  3459 raft_consensus.cc:482] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:01.878994  3459 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:01.880992  3459 raft_consensus.cc:504] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.881794  3459 leader_election.cc:288] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:01.882246  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:01.882469  2412 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:01.882488  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:01.882742  2672 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:01.884469  2412 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:01.884732  2672 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:01.884953  2214 leader_election.cc:302] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:01.885303  3459 raft_consensus.cc:2781] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:01.885789  3459 raft_consensus.cc:686] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:01.886160  3459 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.889705  2139 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:01.949443  3459 raft_consensus.cc:482] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:01.949743  3459 raft_consensus.cc:504] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.950517  3459 leader_election.cc:288] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:01.951236  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:01.951572  2672 raft_consensus.cc:2454] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:01.952009  2216 leader_election.cc:302] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:01.952353  3459 raft_consensus.cc:2781] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:01.952484  3459 raft_consensus.cc:482] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:01.952622  3459 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:01.955977  3459 raft_consensus.cc:504] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.957053  3459 leader_election.cc:288] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:01.957587  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:01.957832  2672 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:01.958791  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:01.958940  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:01.959030  2541 raft_consensus.cc:2454] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:01.960139  2672 raft_consensus.cc:2454] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:01.960626  2216 leader_election.cc:302] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:15:01.961019  3459 raft_consensus.cc:2781] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:01.961194  3459 raft_consensus.cc:686] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:01.961575  3459 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:01.964406  2139 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:02.075229  3317 raft_consensus.cc:482] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:02.075430  3317 raft_consensus.cc:504] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:02.076225  3317 leader_election.cc:288] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:02.076823  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:02.076846  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:02.077071  2672 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:02.077111  2281 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:02.077592  2347 leader_election.cc:302] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:02.077914  3317 raft_consensus.cc:2781] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:02.078039  3317 raft_consensus.cc:482] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:02.078137  3317 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:02.080130  3317 raft_consensus.cc:504] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:02.080776  3317 leader_election.cc:288] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:02.081511  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:02.081496  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:02.081768  2672 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:02.081768  2281 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:02.084697  2672 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:02.084697  2281 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:02.085273  2347 leader_election.cc:302] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:02.085670  3317 raft_consensus.cc:2781] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:02.086091  3317 raft_consensus.cc:686] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:02.086498  3317 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:02.089609  2139 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:15:02.299773  3459 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:02.309727  3477 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:02.401028  3459 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:02.405373  2542 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:02.408577  3459 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:02.516353  3317 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:02.521397  3496 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:02.569380  3485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:02.569909  3485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:02.578271  3485 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 198.061 ms
  time per row: 0.0990303 ms
I20240503 02:15:02.949137  2134 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:02.960814  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 75ba33132e784f408f3e26882aea3d15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.961198  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 75ba33132e784f408f3e26882aea3d15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.961319  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ba33132e784f408f3e26882aea3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.961671  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ba33132e784f408f3e26882aea3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.961505  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 75ba33132e784f408f3e26882aea3d15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.962241  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ba33132e784f408f3e26882aea3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.962714  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 9b822c691d824b2ba25e967535c9734e (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.963114  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.970479  2390 tablet_service.cc:1467] Processing CreateTablet for tablet 9b822c691d824b2ba25e967535c9734e (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.971134  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.972069  3544 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:02.974125  3544 tablet_bootstrap.cc:654] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:02.979568  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 9b822c691d824b2ba25e967535c9734e (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.980180  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.987185  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 7a62b3ef44e94e92b0feed9c228f5702 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.987498  2522 tablet_service.cc:1467] Processing CreateTablet for tablet 7a62b3ef44e94e92b0feed9c228f5702 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.987839  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a62b3ef44e94e92b0feed9c228f5702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.988003  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a62b3ef44e94e92b0feed9c228f5702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:02.988152  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 7a62b3ef44e94e92b0feed9c228f5702 (DEFAULT_TABLE table=rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:02.988523  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a62b3ef44e94e92b0feed9c228f5702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:03.000485  3547 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:03.003455  3547 tablet_bootstrap.cc:654] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.008211  3545 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:03.009397  3543 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:03.009939  3544 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:03.010181  3544 ts_tablet_manager.cc:1367] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.038s	user 0.011s	sys 0.017s
I20240503 02:15:03.011346  3543 tablet_bootstrap.cc:654] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.011240  3544 raft_consensus.cc:348] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.011873  3545 tablet_bootstrap.cc:654] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.011505  3544 raft_consensus.cc:374] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.013700  3544 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.014247  3544 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.016166  3544 ts_tablet_manager.cc:1398] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240503 02:15:03.016566  3544 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:03.019721  3544 tablet_bootstrap.cc:654] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.019927  3547 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:03.020170  3547 ts_tablet_manager.cc:1367] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.004s
I20240503 02:15:03.020473  3543 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:03.020865  3543 ts_tablet_manager.cc:1367] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240503 02:15:03.021234  3547 raft_consensus.cc:348] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.021596  3545 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:03.021633  3547 raft_consensus.cc:374] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.021816  3545 ts_tablet_manager.cc:1367] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.004s
I20240503 02:15:03.021811  3547 raft_consensus.cc:729] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.022244  3547 consensus_queue.cc:260] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.022855  3545 raft_consensus.cc:348] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.023136  3545 raft_consensus.cc:374] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.023265  3545 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.023633  3545 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.024284  3547 ts_tablet_manager.cc:1398] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:03.024430  3545 ts_tablet_manager.cc:1398] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:15:03.024732  3547 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:03.024766  3545 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:03.026932  3545 tablet_bootstrap.cc:654] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.022048  3543 raft_consensus.cc:348] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.027688  3543 raft_consensus.cc:374] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.027848  3543 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.028168  3543 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.028694  3544 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:03.028946  3544 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.007s
I20240503 02:15:03.029891  3544 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.030215  3544 raft_consensus.cc:374] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.030340  3544 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.030683  3544 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.031136  3543 ts_tablet_manager.cc:1398] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240503 02:15:03.031514  3543 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:03.032096  3544 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:03.033535  3543 tablet_bootstrap.cc:654] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.040000  3547 tablet_bootstrap.cc:654] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.043342  3543 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:03.043566  3543 ts_tablet_manager.cc:1367] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240503 02:15:03.043920  3545 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:03.044104  3545 ts_tablet_manager.cc:1367] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.014s
I20240503 02:15:03.044191  3547 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:03.044462  3547 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.020s	user 0.000s	sys 0.006s
I20240503 02:15:03.044473  3543 raft_consensus.cc:348] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.044734  3543 raft_consensus.cc:374] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.044816  3543 raft_consensus.cc:729] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.044937  3545 raft_consensus.cc:348] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.045171  3545 raft_consensus.cc:374] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.045084  3543 consensus_queue.cc:260] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.045286  3545 raft_consensus.cc:729] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.045289  3547 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.045603  3547 raft_consensus.cc:374] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.045701  3547 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.045722  3543 ts_tablet_manager.cc:1398] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:03.045634  3545 consensus_queue.cc:260] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.046000  3547 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.046347  3545 ts_tablet_manager.cc:1398] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:03.047168  3543 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:03.048286  3547 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:15:03.049319  3543 tablet_bootstrap.cc:654] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:03.054097  3543 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:03.054527  3543 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240503 02:15:03.055305  3543 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.055519  3543 raft_consensus.cc:374] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:03.055619  3543 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:03.055838  3543 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.056298  3543 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:03.093986  3496 raft_consensus.cc:482] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:03.094272  3496 raft_consensus.cc:504] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.095032  3496 leader_election.cc:288] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.095702  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:03.095734  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:03.095934  2673 raft_consensus.cc:2454] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:03.095949  2542 raft_consensus.cc:2454] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:03.096444  2347 leader_election.cc:302] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:03.096793  3496 raft_consensus.cc:2781] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:03.096877  3496 raft_consensus.cc:482] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:03.096977  3496 raft_consensus.cc:3037] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.099354  3496 raft_consensus.cc:504] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.100579  3496 leader_election.cc:288] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.100859  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:03.101073  2542 raft_consensus.cc:3037] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.101024  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:03.101250  2673 raft_consensus.cc:3037] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.103224  2542 raft_consensus.cc:2454] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:03.103397  2673 raft_consensus.cc:2454] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:03.103744  2346 leader_election.cc:302] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:03.104094  3496 raft_consensus.cc:2781] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:03.104314  3496 raft_consensus.cc:686] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:03.104712  3496 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.108757  3414 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:03.108565  2131 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:03.108950  3414 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.109038  3459 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:03.109230  3459 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.109855  3414 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.109975  3459 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.110579  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:03.110867  2673 raft_consensus.cc:2454] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:03.111430  2216 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:03.111397  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:03.111658  2281 raft_consensus.cc:2454] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:03.112167  3459 raft_consensus.cc:2781] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:03.112305  3459 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:03.112413  3459 raft_consensus.cc:3037] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.115643  3459 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.116482  3459 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.116923  2478 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:03.117259  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:03.117460  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:03.117642  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:03.117741  2673 raft_consensus.cc:3037] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.118068  2542 raft_consensus.cc:2454] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:03.118719  3414 raft_consensus.cc:2781] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:03.118832  3414 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:03.118917  3414 raft_consensus.cc:3037] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.120424  2673 raft_consensus.cc:2454] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:03.120976  2216 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:15:03.121105  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:03.121305  3459 raft_consensus.cc:2781] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:03.121459  2673 raft_consensus.cc:2379] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08bef4c144bc4e4d8cb5836aee8da852 in current term 1: Already voted for candidate 08055479ea104aa290d19955d3720850 in this term.
I20240503 02:15:03.121613  3459 raft_consensus.cc:686] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:03.121614  3414 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.121891  3459 consensus_queue.cc:237] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:03.122332  3414 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:03.122913  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:03.122911  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:03.123869  2673 raft_consensus.cc:2379] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 08bef4c144bc4e4d8cb5836aee8da852 in current term 1: Already voted for candidate 08055479ea104aa290d19955d3720850 in this term.
I20240503 02:15:03.124290  2477 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852; no voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83
I20240503 02:15:03.124584  3414 raft_consensus.cc:2726] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:15:03.125077  2131 catalog_manager.cc:5484] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:03.153412  3459 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:03.153673  3459 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.154500  3459 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:03.155151  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:03.155201  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:03.155433  2412 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:03.155483  2542 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:03.155946  2214 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:03.156255  3459 raft_consensus.cc:2781] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:03.156351  3459 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:03.156477  3459 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.158951  3459 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.159711  3459 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:03.160271  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:03.160394  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:03.160519  2542 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.160666  2412 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:03.162731  2542 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:03.162962  2412 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:03.163224  2215 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:03.163612  3459 raft_consensus.cc:2781] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:03.163775  3459 raft_consensus.cc:686] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:03.164129  3459 consensus_queue.cc:237] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:03.167322  2131 catalog_manager.cc:5484] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:03.520579  3459 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:03.526952  3488 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:03.597085  3496 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:03.605218  3572 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:03.612648  3561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:03.613006  3561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:03.621038  3561 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 02:15:03.677959  3488 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:03.683492  3459 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:03.882855  2215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 50795.1us

INSERT report
    rows total: 2000
    time total: 269.927 ms
  time per row: 0.134963 ms
I20240503 02:15:04.052109  2131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:04.061795  2521 tablet_service.cc:1467] Processing CreateTablet for tablet a7ab8b0c8db54565a6066b9a7d1569d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.061813  2390 tablet_service.cc:1467] Processing CreateTablet for tablet a7ab8b0c8db54565a6066b9a7d1569d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.062021  2260 tablet_service.cc:1467] Processing CreateTablet for tablet a7ab8b0c8db54565a6066b9a7d1569d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.062256  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ab8b0c8db54565a6066b9a7d1569d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.062281  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ab8b0c8db54565a6066b9a7d1569d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.062487  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ab8b0c8db54565a6066b9a7d1569d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.066694  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 771a4190142b4fd5a9479417122c0881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.067015  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 771a4190142b4fd5a9479417122c0881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.067176  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.067477  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.070505  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 771a4190142b4fd5a9479417122c0881 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.070950  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.074062  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 43e248f0cb764d229802851c4e73cc96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.074491  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e248f0cb764d229802851c4e73cc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.078639  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 43e248f0cb764d229802851c4e73cc96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.079272  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e248f0cb764d229802851c4e73cc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.083197  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 43e248f0cb764d229802851c4e73cc96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:04.083700  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e248f0cb764d229802851c4e73cc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:04.086764  3626 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:04.088189  3625 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:04.089449  3626 tablet_bootstrap.cc:654] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.091089  3625 tablet_bootstrap.cc:654] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.102402  3627 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:04.102943  3624 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:04.103106  3625 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:04.103157  3626 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:04.103310  3625 ts_tablet_manager.cc:1367] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.006s
I20240503 02:15:04.103961  3626 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.000s
I20240503 02:15:04.104569  3627 tablet_bootstrap.cc:654] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.104795  3624 tablet_bootstrap.cc:654] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.104960  3626 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.105814  3626 raft_consensus.cc:374] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.106020  3626 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.106261  3625 raft_consensus.cc:348] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.106527  3626 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.106739  3625 raft_consensus.cc:374] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.106949  3625 raft_consensus.cc:729] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.107299  3625 consensus_queue.cc:260] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.108588  3626 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:04.108783  3625 ts_tablet_manager.cc:1398] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240503 02:15:04.109479  3626 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:04.112747  3626 tablet_bootstrap.cc:654] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.109115  3625 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:04.115790  3624 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:04.115945  3624 ts_tablet_manager.cc:1367] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.013s	user 0.012s	sys 0.000s
I20240503 02:15:04.116024  3627 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:04.116185  3627 ts_tablet_manager.cc:1367] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.009s
I20240503 02:15:04.116829  3627 raft_consensus.cc:348] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.116833  3624 raft_consensus.cc:348] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.117010  3627 raft_consensus.cc:374] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.117020  3624 raft_consensus.cc:374] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.117102  3627 raft_consensus.cc:729] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.117107  3624 raft_consensus.cc:729] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.117445  3625 tablet_bootstrap.cc:654] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.117358  3627 consensus_queue.cc:260] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.117383  3624 consensus_queue.cc:260] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.117966  3624 ts_tablet_manager.cc:1398] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:15:04.118026  3627 ts_tablet_manager.cc:1398] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:04.118295  3627 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:04.119172  3624 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:04.119895  3627 tablet_bootstrap.cc:654] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.120254  3496 raft_consensus.cc:482] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:04.120472  3496 raft_consensus.cc:504] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.121268  3496 leader_election.cc:288] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:04.121402  3624 tablet_bootstrap.cc:654] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.122290  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ab8b0c8db54565a6066b9a7d1569d8" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:04.122593  2541 raft_consensus.cc:2454] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:04.123193  3626 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:04.123420  3626 ts_tablet_manager.cc:1367] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.000s
I20240503 02:15:04.123574  3625 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:04.123610  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ab8b0c8db54565a6066b9a7d1569d8" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:04.123735  3625 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.003s
I20240503 02:15:04.124157  2346 leader_election.cc:302] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:04.124512  3625 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.124728  3625 raft_consensus.cc:374] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.124554  3626 raft_consensus.cc:348] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.124806  3625 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.124858  3626 raft_consensus.cc:374] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.124955  3626 raft_consensus.cc:729] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.125073  3625 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.125463  3483 raft_consensus.cc:2781] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:04.125303  3626 consensus_queue.cc:260] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.125558  3483 raft_consensus.cc:482] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:04.125659  3483 raft_consensus.cc:3037] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.125730  3625 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:04.126082  3625 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:04.127923  3626 ts_tablet_manager.cc:1398] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
W20240503 02:15:04.125208  2348 leader_election.cc:341] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:15:04.130587  3627 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:04.130827  3627 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.005s
I20240503 02:15:04.131651  3483 raft_consensus.cc:504] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.131835  3627 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.132067  3627 raft_consensus.cc:374] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.132136  3627 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.132372  3627 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.132894  3483 leader_election.cc:288] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:04.133088  3627 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:04.133144  3625 tablet_bootstrap.cc:654] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:04.133188  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ab8b0c8db54565a6066b9a7d1569d8" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:04.133466  2541 raft_consensus.cc:3037] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.133852  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7ab8b0c8db54565a6066b9a7d1569d8" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:04.134079  2281 raft_consensus.cc:3037] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.136953  2281 raft_consensus.cc:2454] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:04.137589  2348 leader_election.cc:302] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:04.137851  3624 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:04.137972  3483 raft_consensus.cc:2781] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:04.138033  3624 ts_tablet_manager.cc:1367] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.007s
I20240503 02:15:04.138171  3483 raft_consensus.cc:686] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:04.138543  3483 consensus_queue.cc:237] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.139556  3624 raft_consensus.cc:348] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.139791  3624 raft_consensus.cc:374] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.139873  3624 raft_consensus.cc:729] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.140146  3624 consensus_queue.cc:260] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.141420  3624 ts_tablet_manager.cc:1398] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:15:04.142232  2541 raft_consensus.cc:2454] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:04.143836  3625 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:04.143997  3625 ts_tablet_manager.cc:1367] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.018s	user 0.002s	sys 0.009s
I20240503 02:15:04.144991  3625 raft_consensus.cc:348] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.145202  3625 raft_consensus.cc:374] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:04.144991  2131 catalog_manager.cc:5484] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:04.145291  3625 raft_consensus.cc:729] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:04.145553  3625 consensus_queue.cc:260] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.146086  3625 ts_tablet_manager.cc:1398] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:04.331012  3483 raft_consensus.cc:482] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:04.331225  3483 raft_consensus.cc:504] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.332116  3483 leader_election.cc:288] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:04.332615  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:04.332679  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:04.332857  2673 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:04.332890  2281 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:04.333305  2348 leader_election.cc:302] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:04.333648  3483 raft_consensus.cc:2781] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:04.333745  3483 raft_consensus.cc:482] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:04.333808  3483 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.335991  3483 raft_consensus.cc:504] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.336586  3483 leader_election.cc:288] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:04.337072  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:04.337112  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:04.337356  2281 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.337356  2673 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.340266  2281 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:04.340266  2673 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:04.340744  2347 leader_election.cc:302] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:04.341097  3483 raft_consensus.cc:2781] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:04.341238  3483 raft_consensus.cc:686] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:04.341533  3483 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:04.344383  2131 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:04.362538  3397 raft_consensus.cc:482] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:04.363008  3397 raft_consensus.cc:504] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.363732  3397 leader_election.cc:288] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:04.364305  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e248f0cb764d229802851c4e73cc96" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:04.364547  2541 raft_consensus.cc:2454] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:04.364621  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e248f0cb764d229802851c4e73cc96" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:04.364970  2607 leader_election.cc:302] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:04.364972  2412 raft_consensus.cc:2454] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:04.365295  3397 raft_consensus.cc:2781] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:04.365440  3397 raft_consensus.cc:482] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:04.365582  3397 raft_consensus.cc:3037] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.368103  3397 raft_consensus.cc:504] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.368790  3397 leader_election.cc:288] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:04.369400  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e248f0cb764d229802851c4e73cc96" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:04.369400  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e248f0cb764d229802851c4e73cc96" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:04.369601  2541 raft_consensus.cc:3037] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.369613  2412 raft_consensus.cc:3037] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:04.371520  2412 raft_consensus.cc:2454] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:04.371551  2541 raft_consensus.cc:2454] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:04.372026  2607 leader_election.cc:302] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:04.372355  3397 raft_consensus.cc:2781] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:04.372736  3397 raft_consensus.cc:686] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:15:04.373000  3397 consensus_queue.cc:237] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:04.376118  2131 catalog_manager.cc:5484] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:15:04.534170  3483 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:04.540918  3496 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:04.779031  3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:04.779376  3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:04.788223  3641 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:15:04.839588  3496 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:04.845561  3483 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:04.855831  3397 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:04.860550  3639 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 226.764 ms
  time per row: 0.113382 ms
I20240503 02:15:05.175676  2132 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:05.185963  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 586ebf467f0f427fa62dee6223a39848 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.186036  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 586ebf467f0f427fa62dee6223a39848 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.186187  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 586ebf467f0f427fa62dee6223a39848 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.186486  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586ebf467f0f427fa62dee6223a39848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.186503  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586ebf467f0f427fa62dee6223a39848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.186580  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586ebf467f0f427fa62dee6223a39848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.188998  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7c9c29384949fb95cb0c3b765f57b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.189004  2522 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7c9c29384949fb95cb0c3b765f57b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.189461  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7c9c29384949fb95cb0c3b765f57b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.189585  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7c9c29384949fb95cb0c3b765f57b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.189762  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 913d0aa014ad4bcaac905b5fdaa296d2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.190138  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d0aa014ad4bcaac905b5fdaa296d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.194162  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 913d0aa014ad4bcaac905b5fdaa296d2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.194634  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d0aa014ad4bcaac905b5fdaa296d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.195645  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 913d0aa014ad4bcaac905b5fdaa296d2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.196192  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d0aa014ad4bcaac905b5fdaa296d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.199393  2259 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7c9c29384949fb95cb0c3b765f57b0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:05.200002  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7c9c29384949fb95cb0c3b765f57b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:05.214490  3705 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:05.217458  3705 tablet_bootstrap.cc:654] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.229243  3706 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:05.230008  3707 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:05.232744  3706 tablet_bootstrap.cc:654] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.233492  3704 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:05.233808  3707 tablet_bootstrap.cc:654] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.235463  3704 tablet_bootstrap.cc:654] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.236738  3705 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:05.236922  3705 ts_tablet_manager.cc:1367] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.023s	user 0.000s	sys 0.010s
I20240503 02:15:05.237861  3705 raft_consensus.cc:348] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.238142  3705 raft_consensus.cc:374] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.238282  3705 raft_consensus.cc:729] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.238588  3705 consensus_queue.cc:260] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.239292  3705 ts_tablet_manager.cc:1398] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:15:05.239706  3707 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:05.239928  3707 ts_tablet_manager.cc:1367] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.000s
I20240503 02:15:05.240605  3704 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:05.240756  3707 raft_consensus.cc:348] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.240962  3707 raft_consensus.cc:374] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.240898  3704 ts_tablet_manager.cc:1367] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240503 02:15:05.241071  3707 raft_consensus.cc:729] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.241312  3707 consensus_queue.cc:260] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.242141  3707 ts_tablet_manager.cc:1398] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:15:05.242542  3704 raft_consensus.cc:348] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.242755  3704 raft_consensus.cc:374] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.243058  3707 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:05.243861  3704 raft_consensus.cc:729] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.244529  3704 consensus_queue.cc:260] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.245033  3707 tablet_bootstrap.cc:654] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.246448  3706 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:05.246691  3706 ts_tablet_manager.cc:1367] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.018s	user 0.004s	sys 0.005s
I20240503 02:15:05.247403  3704 ts_tablet_manager.cc:1398] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20240503 02:15:05.247650  3706 raft_consensus.cc:348] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.247797  3704 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:05.247874  3706 raft_consensus.cc:374] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.247987  3706 raft_consensus.cc:729] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.248222  3706 consensus_queue.cc:260] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.248826  3706 ts_tablet_manager.cc:1398] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:05.249125  3706 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:05.250360  3707 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:05.250617  3707 ts_tablet_manager.cc:1367] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.004s
I20240503 02:15:05.251044  3706 tablet_bootstrap.cc:654] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.251369  3704 tablet_bootstrap.cc:654] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.251559  3707 raft_consensus.cc:348] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.251849  3707 raft_consensus.cc:374] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.251961  3707 raft_consensus.cc:729] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.252302  3707 consensus_queue.cc:260] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.253502  3609 raft_consensus.cc:482] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:05.253651  3609 raft_consensus.cc:504] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.254377  3707 ts_tablet_manager.cc:1398] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:15:05.254405  3609 leader_election.cc:288] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:05.254768  3707 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:05.255709  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:05.256465  3707 tablet_bootstrap.cc:654] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
W20240503 02:15:05.256726  2215 leader_election.cc:341] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:15:05.260819  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
W20240503 02:15:05.261787  2216 leader_election.cc:341] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:15:05.261973  2216 leader_election.cc:302] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08055479ea104aa290d19955d3720850; no voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83
I20240503 02:15:05.262470  3609 raft_consensus.cc:2726] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:15:05.263032  3706 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:05.263202  3706 ts_tablet_manager.cc:1367] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240503 02:15:05.263823  3706 raft_consensus.cc:348] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.264071  3706 raft_consensus.cc:374] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.264153  3706 raft_consensus.cc:729] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.264406  3706 consensus_queue.cc:260] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.264878  3706 ts_tablet_manager.cc:1398] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:05.266310  3704 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:05.266340  3707 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:05.266551  3707 ts_tablet_manager.cc:1367] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.000s
I20240503 02:15:05.266551  3704 ts_tablet_manager.cc:1367] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.001s
I20240503 02:15:05.267429  3704 raft_consensus.cc:348] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.267696  3704 raft_consensus.cc:374] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.267815  3704 raft_consensus.cc:729] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.267716  3707 raft_consensus.cc:348] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.268117  3704 consensus_queue.cc:260] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.268260  3707 raft_consensus.cc:374] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.268448  3707 raft_consensus.cc:729] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.268790  3704 ts_tablet_manager.cc:1398] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:05.269131  3704 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:05.270987  3704 tablet_bootstrap.cc:654] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:05.268779  3707 consensus_queue.cc:260] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.271796  3707 ts_tablet_manager.cc:1398] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:15:05.274842  3704 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:05.275108  3704 ts_tablet_manager.cc:1367] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.002s
I20240503 02:15:05.275812  3704 raft_consensus.cc:348] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.276083  3704 raft_consensus.cc:374] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:05.276186  3704 raft_consensus.cc:729] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:05.276445  3704 consensus_queue.cc:260] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.277006  3704 ts_tablet_manager.cc:1398] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:05.341029  3647 raft_consensus.cc:482] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:05.341207  3647 raft_consensus.cc:504] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.342057  3647 leader_election.cc:288] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:05.342589  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:05.342613  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:05.342828  2673 raft_consensus.cc:2454] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:05.343065  2280 raft_consensus.cc:2454] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:05.343407  2477 leader_election.cc:302] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:05.343725  3647 raft_consensus.cc:2781] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:05.343863  3647 raft_consensus.cc:482] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:05.343979  3647 raft_consensus.cc:3037] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.346120  3647 raft_consensus.cc:504] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.346735  3647 leader_election.cc:288] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:05.347522  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:05.347501  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913d0aa014ad4bcaac905b5fdaa296d2" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:05.347800  2280 raft_consensus.cc:3037] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.347831  2673 raft_consensus.cc:3037] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.351220  2280 raft_consensus.cc:2454] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:05.351209  2673 raft_consensus.cc:2454] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:05.351823  2478 leader_election.cc:302] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:05.352203  3647 raft_consensus.cc:2781] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:05.352720  3647 raft_consensus.cc:686] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:05.353077  3647 consensus_queue.cc:237] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:05.356164  2135 catalog_manager.cc:5484] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:05.444501  3715 raft_consensus.cc:482] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:05.444825  3715 raft_consensus.cc:504] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.445639  3715 leader_election.cc:288] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:05.446352  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586ebf467f0f427fa62dee6223a39848" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:05.446580  2673 raft_consensus.cc:2454] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:05.446539  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586ebf467f0f427fa62dee6223a39848" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:05.446980  2541 raft_consensus.cc:2454] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:05.447146  2216 leader_election.cc:302] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:05.447446  3715 raft_consensus.cc:2781] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:05.447548  3715 raft_consensus.cc:482] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:05.447644  3715 raft_consensus.cc:3037] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.449838  3715 raft_consensus.cc:504] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.450472  3715 leader_election.cc:288] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:05.450842  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586ebf467f0f427fa62dee6223a39848" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:05.451025  2673 raft_consensus.cc:3037] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.451223  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "586ebf467f0f427fa62dee6223a39848" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:05.451440  2541 raft_consensus.cc:3037] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.454156  2673 raft_consensus.cc:2454] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:05.454207  2541 raft_consensus.cc:2454] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:05.454658  3496 raft_consensus.cc:482] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:05.454798  2215 leader_election.cc:302] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:05.454902  3496 raft_consensus.cc:504] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.455477  3715 raft_consensus.cc:2781] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:05.455567  3496 leader_election.cc:288] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:05.455637  3715 raft_consensus.cc:686] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:05.456144  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7c9c29384949fb95cb0c3b765f57b0" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:05.456015  3715 consensus_queue.cc:237] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.456167  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7c9c29384949fb95cb0c3b765f57b0" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:05.456326  2541 raft_consensus.cc:2454] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:05.456431  2280 raft_consensus.cc:2454] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:05.456725  2346 leader_election.cc:302] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:05.457019  3496 raft_consensus.cc:2781] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:05.457126  3496 raft_consensus.cc:482] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:05.457227  3496 raft_consensus.cc:3037] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.459610  3496 raft_consensus.cc:504] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.460543  3496 leader_election.cc:288] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:05.461097  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7c9c29384949fb95cb0c3b765f57b0" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:05.461268  2541 raft_consensus.cc:3037] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.461555  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7c9c29384949fb95cb0c3b765f57b0" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:05.461475  2135 catalog_manager.cc:5484] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:05.461786  2280 raft_consensus.cc:3037] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:05.463536  2541 raft_consensus.cc:2454] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:05.463991  2346 leader_election.cc:302] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:05.464288  3496 raft_consensus.cc:2781] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:05.464284  2280 raft_consensus.cc:2454] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:05.464401  3496 raft_consensus.cc:686] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:05.464677  3496 consensus_queue.cc:237] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:05.469206  2137 catalog_manager.cc:5484] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:05.756850  3647 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:05.780376  3647 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:05.939872  3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:05.940305  3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:15:05.947538  3496 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:05.949975  3722 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:05.953275  3496 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 02:15:06.067875  3715 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:06.071596  3715 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 313.311 ms
  time per row: 0.156656 ms
I20240503 02:15:06.431001  2136 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:06.441051  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 62a3dcc2de594c8982477a0229f71191 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.441053  2653 tablet_service.cc:1467] Processing CreateTablet for tablet 62a3dcc2de594c8982477a0229f71191 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.441648  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.441884  2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.442337  2521 tablet_service.cc:1467] Processing CreateTablet for tablet a1949a5c7cc741b19da021eccb7d0dd1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.442606  2259 tablet_service.cc:1467] Processing CreateTablet for tablet 62a3dcc2de594c8982477a0229f71191 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.442740  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1949a5c7cc741b19da021eccb7d0dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.443030  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.445896  2651 tablet_service.cc:1467] Processing CreateTablet for tablet a1949a5c7cc741b19da021eccb7d0dd1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.446449  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1949a5c7cc741b19da021eccb7d0dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.454721  3789 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:06.454973  2390 tablet_service.cc:1467] Processing CreateTablet for tablet a1949a5c7cc741b19da021eccb7d0dd1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.455086  2652 tablet_service.cc:1467] Processing CreateTablet for tablet 02e0b954008c4f30af414da6cb5d59c2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.455613  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1949a5c7cc741b19da021eccb7d0dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.455627  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.456334  3787 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:06.457382  3789 tablet_bootstrap.cc:654] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.458662  3787 tablet_bootstrap.cc:654] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.466063  2392 tablet_service.cc:1467] Processing CreateTablet for tablet 02e0b954008c4f30af414da6cb5d59c2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.466687  2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.468470  3789 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:06.468722  3789 ts_tablet_manager.cc:1367] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.000s
I20240503 02:15:06.468824  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 02e0b954008c4f30af414da6cb5d59c2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:06.469301  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:06.469707  3789 raft_consensus.cc:348] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.470356  3789 raft_consensus.cc:374] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.470453  3789 raft_consensus.cc:729] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.471043  3789 consensus_queue.cc:260] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.471501  3787 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:06.471649  3787 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.010s
I20240503 02:15:06.472467  3787 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.472664  3787 raft_consensus.cc:374] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.472744  3787 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.472999  3787 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.474768  3787 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:15:06.475054  3787 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:06.475878  3786 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:06.476876  3787 tablet_bootstrap.cc:654] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.477756  3786 tablet_bootstrap.cc:654] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.483021  3788 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:06.483052  3789 ts_tablet_manager.cc:1398] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.000s
I20240503 02:15:06.484839  3786 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:06.485023  3786 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240503 02:15:06.486017  3786 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.486423  3786 raft_consensus.cc:374] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.486574  3786 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.486877  3786 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.487217  3787 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:06.487493  3787 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240503 02:15:06.487716  3786 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:15:06.487862  3788 tablet_bootstrap.cc:654] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.488157  3786 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:06.488834  3787 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.489233  3787 raft_consensus.cc:374] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.489461  3787 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.490075  3787 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.490263  3786 tablet_bootstrap.cc:654] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.491003  3787 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:15:06.491428  3787 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:06.493188  3787 tablet_bootstrap.cc:654] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.496847  3788 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:06.497102  3788 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.007s
I20240503 02:15:06.498324  3788 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.498647  3788 raft_consensus.cc:374] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.498772  3788 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.499184  3788 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.500165  3788 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:15:06.500602  3788 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:06.502209  3787 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:06.502450  3787 ts_tablet_manager.cc:1367] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240503 02:15:06.503455  3787 raft_consensus.cc:348] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.503762  3787 raft_consensus.cc:374] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.503908  3787 raft_consensus.cc:729] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.504310  3787 consensus_queue.cc:260] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.505223  3787 ts_tablet_manager.cc:1398] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:15:06.505635  3788 tablet_bootstrap.cc:654] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.506850  3786 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:06.507237  3786 ts_tablet_manager.cc:1367] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.010s
I20240503 02:15:06.507989  3786 raft_consensus.cc:348] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.508219  3786 raft_consensus.cc:374] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.508319  3788 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:06.508306  3786 raft_consensus.cc:729] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.508493  3788 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.004s
I20240503 02:15:06.508575  3786 consensus_queue.cc:260] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.509192  3786 ts_tablet_manager.cc:1398] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:06.509423  3788 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.509594  3786 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:06.509650  3788 raft_consensus.cc:374] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.509716  3788 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.510026  3788 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.510671  3788 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:06.511612  3786 tablet_bootstrap.cc:654] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:06.514230  3786 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:06.514400  3786 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240503 02:15:06.515249  3786 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.515460  3786 raft_consensus.cc:374] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:06.515558  3786 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:06.515769  3786 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.516280  3786 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:06.555680  3754 raft_consensus.cc:482] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:06.555871  3754 raft_consensus.cc:504] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.556582  3754 leader_election.cc:288] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:06.557186  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e0b954008c4f30af414da6cb5d59c2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:06.557201  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e0b954008c4f30af414da6cb5d59c2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:06.557463  2672 raft_consensus.cc:2454] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:06.557526  2411 raft_consensus.cc:2454] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:06.557963  2216 leader_election.cc:302] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:06.558255  3754 raft_consensus.cc:2781] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:06.558364  3754 raft_consensus.cc:482] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:06.558480  3754 raft_consensus.cc:3037] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.560690  3754 raft_consensus.cc:504] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.561422  3754 leader_election.cc:288] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:06.562268  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e0b954008c4f30af414da6cb5d59c2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:06.562284  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e0b954008c4f30af414da6cb5d59c2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:06.562507  2672 raft_consensus.cc:3037] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.562543  2411 raft_consensus.cc:3037] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.565610  2411 raft_consensus.cc:2454] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:06.565559  2672 raft_consensus.cc:2454] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:06.566273  2214 leader_election.cc:302] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:06.566659  3754 raft_consensus.cc:2781] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:06.566840  3754 raft_consensus.cc:686] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:06.567138  3754 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.567543  3639 raft_consensus.cc:482] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:06.567889  3639 raft_consensus.cc:504] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.568631  3639 leader_election.cc:288] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:06.569164  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1949a5c7cc741b19da021eccb7d0dd1" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:06.569416  2542 raft_consensus.cc:2454] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:06.569855  2607 leader_election.cc:302] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:06.569926  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1949a5c7cc741b19da021eccb7d0dd1" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:06.570190  3639 raft_consensus.cc:2781] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:06.570225  2411 raft_consensus.cc:2454] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:06.570291  3639 raft_consensus.cc:482] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:06.570408  3639 raft_consensus.cc:3037] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.571996  2132 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:06.572899  3639 raft_consensus.cc:504] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.573648  3639 leader_election.cc:288] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:06.574129  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1949a5c7cc741b19da021eccb7d0dd1" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:06.574141  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1949a5c7cc741b19da021eccb7d0dd1" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:06.574344  2411 raft_consensus.cc:3037] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.574347  2542 raft_consensus.cc:3037] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.576615  2411 raft_consensus.cc:2454] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:06.576617  2542 raft_consensus.cc:2454] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:06.577095  2607 leader_election.cc:302] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:06.577461  3639 raft_consensus.cc:2781] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:06.577880  3639 raft_consensus.cc:686] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:15:06.578210  3639 consensus_queue.cc:237] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:06.581452  2132 catalog_manager.cc:5484] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:15:06.601918  3639 raft_consensus.cc:482] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:06.602149  3639 raft_consensus.cc:504] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.602862  3639 leader_election.cc:288] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:06.603649  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:06.603616  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:06.603947  2411 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:06.603930  2281 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:06.604521  2606 leader_election.cc:302] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:06.604856  3639 raft_consensus.cc:2781] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:06.604981  3639 raft_consensus.cc:482] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:06.605139  3639 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.607333  3639 raft_consensus.cc:504] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.607841  3639 leader_election.cc:288] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:06.608557  2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:06.608709  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:06.608870  2411 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.608944  2281 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:06.610982  2281 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:06.610993  2411 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:06.611543  2609 leader_election.cc:302] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:06.611989  3639 raft_consensus.cc:2781] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:06.612175  3639 raft_consensus.cc:686] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:15:06.612489  3639 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:06.615774  2132 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:06.958292  3639 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:06.965103  3808 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:06.970320  3715 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:06.983419  3715 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:07.052055  3803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:07.052454  3803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:07.060942  3803 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 02:15:07.187788  3808 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:07.198695  3639 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 299.08 ms
  time per row: 0.14954 ms
I20240503 02:15:07.541078  2132 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:07.551084  2391 tablet_service.cc:1467] Processing CreateTablet for tablet a6e0d39087944e4ea13a17ca4af478ab (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.551378  2521 tablet_service.cc:1467] Processing CreateTablet for tablet a6e0d39087944e4ea13a17ca4af478ab (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.551607  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e0d39087944e4ea13a17ca4af478ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.551765  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e0d39087944e4ea13a17ca4af478ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.552016  2392 tablet_service.cc:1467] Processing CreateTablet for tablet 592cf4d5b35e49bfa4173343d64367f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.552346  2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.555891  2259 tablet_service.cc:1467] Processing CreateTablet for tablet a6e0d39087944e4ea13a17ca4af478ab (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.556401  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e0d39087944e4ea13a17ca4af478ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.557476  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 592cf4d5b35e49bfa4173343d64367f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.557909  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.561957  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 592cf4d5b35e49bfa4173343d64367f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.562872  2652 tablet_service.cc:1467] Processing CreateTablet for tablet ae239d73f0bc4d9aac025e40489e81a6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.564170  2522 tablet_service.cc:1467] Processing CreateTablet for tablet ae239d73f0bc4d9aac025e40489e81a6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.564741  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae239d73f0bc4d9aac025e40489e81a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.568032  2260 tablet_service.cc:1467] Processing CreateTablet for tablet ae239d73f0bc4d9aac025e40489e81a6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:07.568717  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae239d73f0bc4d9aac025e40489e81a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.571877  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.571895  3865 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:07.573200  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae239d73f0bc4d9aac025e40489e81a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:07.574791  3865 tablet_bootstrap.cc:654] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.585510  3865 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:07.585840  3865 ts_tablet_manager.cc:1367] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.004s
I20240503 02:15:07.587029  3866 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:07.587924  3865 raft_consensus.cc:348] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.588303  3865 raft_consensus.cc:374] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.588456  3865 raft_consensus.cc:729] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.588461  3869 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:07.588719  3868 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:07.589025  3866 tablet_bootstrap.cc:654] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.589013  3865 consensus_queue.cc:260] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.590888  3865 ts_tablet_manager.cc:1398] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:07.591275  3865 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:07.591276  3868 tablet_bootstrap.cc:654] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.593248  3869 tablet_bootstrap.cc:654] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.594781  3865 tablet_bootstrap.cc:654] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.600271  3869 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:07.600708  3869 ts_tablet_manager.cc:1367] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240503 02:15:07.601953  3869 raft_consensus.cc:348] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.602386  3869 raft_consensus.cc:374] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.602567  3869 raft_consensus.cc:729] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.602965  3869 consensus_queue.cc:260] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.603989  3869 ts_tablet_manager.cc:1398] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:07.604352  3869 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:07.607620  3865 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:07.607901  3865 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.004s
I20240503 02:15:07.607995  3869 tablet_bootstrap.cc:654] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.608825  3865 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.609370  3866 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:07.609599  3866 ts_tablet_manager.cc:1367] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.023s	user 0.007s	sys 0.012s
I20240503 02:15:07.609167  3865 raft_consensus.cc:374] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.609876  3865 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.610240  3865 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.610918  3865 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:15:07.612821  3866 raft_consensus.cc:348] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.612929  3868 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:07.613015  3866 raft_consensus.cc:374] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.613091  3868 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.004s
I20240503 02:15:07.613132  3866 raft_consensus.cc:729] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.613927  3869 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:07.614157  3869 ts_tablet_manager.cc:1367] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.004s
I20240503 02:15:07.614197  3868 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.614537  3868 raft_consensus.cc:374] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.614686  3868 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.614961  3868 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.615036  3869 raft_consensus.cc:348] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.615301  3869 raft_consensus.cc:374] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.615429  3869 raft_consensus.cc:729] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.615715  3868 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:15:07.615734  3869 consensus_queue.cc:260] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.616658  3869 ts_tablet_manager.cc:1398] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:07.615906  3866 consensus_queue.cc:260] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.617409  3868 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:07.617563  3866 ts_tablet_manager.cc:1398] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240503 02:15:07.617887  3866 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:07.619607  3866 tablet_bootstrap.cc:654] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.621295  3868 tablet_bootstrap.cc:654] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.623405  3866 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:07.623687  3866 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.001s
I20240503 02:15:07.624794  3866 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.625159  3866 raft_consensus.cc:374] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.625260  3866 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.625636  3866 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.626013  3868 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:07.626202  3868 ts_tablet_manager.cc:1367] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.001s
I20240503 02:15:07.626385  3866 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:15:07.626827  3866 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:07.626927  3868 raft_consensus.cc:348] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.627200  3868 raft_consensus.cc:374] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.627308  3868 raft_consensus.cc:729] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.627606  3868 consensus_queue.cc:260] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.628192  3868 ts_tablet_manager.cc:1398] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:07.628466  3866 tablet_bootstrap.cc:654] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:07.630582  3866 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:07.630757  3866 ts_tablet_manager.cc:1367] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.001s
I20240503 02:15:07.631690  3866 raft_consensus.cc:348] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.631949  3866 raft_consensus.cc:374] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:07.632064  3866 raft_consensus.cc:729] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:07.632298  3866 consensus_queue.cc:260] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.632977  3866 ts_tablet_manager.cc:1398] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:07.659206  3718 raft_consensus.cc:482] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:07.659409  3718 raft_consensus.cc:504] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.660151  3718 leader_election.cc:288] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:07.660755  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:07.660774  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:07.661028  2410 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:07.661132  2672 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:07.661840  2475 leader_election.cc:302] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:07.662263  3718 raft_consensus.cc:2781] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:07.662401  3718 raft_consensus.cc:482] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:07.662501  3718 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.664498  3718 raft_consensus.cc:504] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.665271  3718 leader_election.cc:288] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:07.665859  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:07.665983  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:07.666072  2672 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.666208  2410 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.668280  2410 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:07.668277  2672 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:07.668787  2475 leader_election.cc:302] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:07.669165  3718 raft_consensus.cc:2781] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:07.669610  3718 raft_consensus.cc:686] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:07.669914  3718 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:07.673110  2132 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:15:07.710537  3718 raft_consensus.cc:482] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:07.710789  3718 raft_consensus.cc:504] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.711472  3718 leader_election.cc:288] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:07.712185  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae239d73f0bc4d9aac025e40489e81a6" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:07.712185  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae239d73f0bc4d9aac025e40489e81a6" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:07.712424  2672 raft_consensus.cc:2454] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:07.712421  2279 raft_consensus.cc:2454] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:07.712898  2477 leader_election.cc:302] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:07.713222  3718 raft_consensus.cc:2781] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:07.713315  3718 raft_consensus.cc:482] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:07.713461  3718 raft_consensus.cc:3037] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.715674  3718 raft_consensus.cc:504] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.716444  3718 leader_election.cc:288] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:07.717286  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae239d73f0bc4d9aac025e40489e81a6" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:07.717286  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae239d73f0bc4d9aac025e40489e81a6" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:07.717563  2672 raft_consensus.cc:3037] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.717563  2279 raft_consensus.cc:3037] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.720839  2279 raft_consensus.cc:2454] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:07.720839  2672 raft_consensus.cc:2454] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:07.721521  2477 leader_election.cc:302] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:07.721954  3718 raft_consensus.cc:2781] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:07.722187  3718 raft_consensus.cc:686] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:07.722512  3718 consensus_queue.cc:237] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.726496  2132 catalog_manager.cc:5484] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:07.796221  3781 raft_consensus.cc:482] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:07.796517  3781 raft_consensus.cc:504] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.797436  3781 leader_election.cc:288] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:07.798017  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e0d39087944e4ea13a17ca4af478ab" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:07.798277  2279 raft_consensus.cc:2454] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:07.798283  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e0d39087944e4ea13a17ca4af478ab" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:07.798578  2542 raft_consensus.cc:2454] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:07.798758  2348 leader_election.cc:302] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:07.799113  3781 raft_consensus.cc:2781] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:07.799237  3781 raft_consensus.cc:482] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:07.799331  3781 raft_consensus.cc:3037] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.801591  3781 raft_consensus.cc:504] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.802228  3781 leader_election.cc:288] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:07.802764  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e0d39087944e4ea13a17ca4af478ab" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:07.802810  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6e0d39087944e4ea13a17ca4af478ab" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:07.802980  2542 raft_consensus.cc:3037] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.803022  2279 raft_consensus.cc:3037] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:07.805006  2542 raft_consensus.cc:2454] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:07.805085  2279 raft_consensus.cc:2454] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:07.805549  2346 leader_election.cc:302] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:07.805910  3781 raft_consensus.cc:2781] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:07.806394  3781 raft_consensus.cc:686] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:07.807016  3781 consensus_queue.cc:237] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:07.810619  2132 catalog_manager.cc:5484] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:08.069752  3718 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:08.079593  3718 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:08.117503  3718 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:08.122457  3890 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:08.262292  3781 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:08.270701  3781 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:08.280961  3883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:08.281389  3883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:08.291558  3883 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'

INSERT report
    rows total: 2000
    time total: 294.181 ms
  time per row: 0.147091 ms
I20240503 02:15:08.766345  2132 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:08.778545  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.778563  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.779131  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dbf0fd54b14c2fa21d3e62104c9b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.779142  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dbf0fd54b14c2fa21d3e62104c9b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.781805  2652 tablet_service.cc:1467] Processing CreateTablet for tablet f0a088d230d0462980648628eae548d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.782301  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a088d230d0462980648628eae548d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.782236  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.782351  2392 tablet_service.cc:1467] Processing CreateTablet for tablet f0a088d230d0462980648628eae548d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.782723  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dbf0fd54b14c2fa21d3e62104c9b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.783016  2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a088d230d0462980648628eae548d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.783361  2522 tablet_service.cc:1467] Processing CreateTablet for tablet eadcb49d12664e6fb5a5ad8dbe47f342 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.783421  2651 tablet_service.cc:1467] Processing CreateTablet for tablet eadcb49d12664e6fb5a5ad8dbe47f342 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.783746  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadcb49d12664e6fb5a5ad8dbe47f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.783792  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadcb49d12664e6fb5a5ad8dbe47f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.789901  2259 tablet_service.cc:1467] Processing CreateTablet for tablet f0a088d230d0462980648628eae548d2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.790309  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a088d230d0462980648628eae548d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.793558  2390 tablet_service.cc:1467] Processing CreateTablet for tablet eadcb49d12664e6fb5a5ad8dbe47f342 (DEFAULT_TABLE table=rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:08.793977  2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadcb49d12664e6fb5a5ad8dbe47f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:08.800907  3945 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:08.804098  3944 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:08.804587  3945 tablet_bootstrap.cc:654] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.804711  3942 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:08.809828  3942 tablet_bootstrap.cc:654] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.810343  3944 tablet_bootstrap.cc:654] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.817716  3946 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:08.819070  3945 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:08.819401  3945 ts_tablet_manager.cc:1367] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.009s
I20240503 02:15:08.820294  3946 tablet_bootstrap.cc:654] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.820549  3945 raft_consensus.cc:348] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.820928  3945 raft_consensus.cc:374] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.821102  3945 raft_consensus.cc:729] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.821444  3945 consensus_queue.cc:260] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.822322  3945 ts_tablet_manager.cc:1398] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:08.822747  3945 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:08.825066  3945 tablet_bootstrap.cc:654] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.830632  3946 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:08.835211  3946 ts_tablet_manager.cc:1367] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.018s	user 0.003s	sys 0.003s
I20240503 02:15:08.836220  3946 raft_consensus.cc:348] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.836585  3946 raft_consensus.cc:374] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.836725  3946 raft_consensus.cc:729] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.837070  3946 consensus_queue.cc:260] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.838780  3944 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:08.839022  3944 ts_tablet_manager.cc:1367] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.035s	user 0.003s	sys 0.005s
I20240503 02:15:08.839215  3946 ts_tablet_manager.cc:1398] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:08.839586  3946 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:08.839967  3944 raft_consensus.cc:348] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.840299  3944 raft_consensus.cc:374] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.840472  3944 raft_consensus.cc:729] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.840840  3944 consensus_queue.cc:260] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.841430  3946 tablet_bootstrap.cc:654] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.841605  3942 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:08.841776  3942 ts_tablet_manager.cc:1367] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.037s	user 0.007s	sys 0.000s
I20240503 02:15:08.842620  3942 raft_consensus.cc:348] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.842852  3942 raft_consensus.cc:374] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.842988  3942 raft_consensus.cc:729] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.843240  3942 consensus_queue.cc:260] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.843878  3942 ts_tablet_manager.cc:1398] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:15:08.844295  3942 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:08.853816  3944 ts_tablet_manager.cc:1398] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.015s	user 0.000s	sys 0.002s
I20240503 02:15:08.855562  3944 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:08.861289  3942 tablet_bootstrap.cc:654] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.860260  3944 tablet_bootstrap.cc:654] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.863296  3945 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:08.863508  3945 ts_tablet_manager.cc:1367] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.041s	user 0.017s	sys 0.009s
I20240503 02:15:08.864500  3945 raft_consensus.cc:348] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.864827  3945 raft_consensus.cc:374] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.864938  3945 raft_consensus.cc:729] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.865270  3945 consensus_queue.cc:260] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.866166  3945 ts_tablet_manager.cc:1398] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:15:08.868289  3942 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:08.868518  3942 ts_tablet_manager.cc:1367] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.024s	user 0.000s	sys 0.006s
I20240503 02:15:08.868968  3944 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:08.869150  3944 ts_tablet_manager.cc:1367] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.006s
I20240503 02:15:08.869459  3942 raft_consensus.cc:348] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.869685  3942 raft_consensus.cc:374] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.869756  3942 raft_consensus.cc:729] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.870275  3942 consensus_queue.cc:260] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.870679  3944 raft_consensus.cc:348] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.870937  3944 raft_consensus.cc:374] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.871165  3944 raft_consensus.cc:729] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.871457  3944 consensus_queue.cc:260] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.872073  3944 ts_tablet_manager.cc:1398] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:15:08.872818  3946 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:08.872848  3942 ts_tablet_manager.cc:1398] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:08.873034  3946 ts_tablet_manager.cc:1367] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.009s
I20240503 02:15:08.873184  3942 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:08.873908  3946 raft_consensus.cc:348] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.874161  3946 raft_consensus.cc:374] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.874238  3946 raft_consensus.cc:729] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.874498  3946 consensus_queue.cc:260] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.875120  3946 ts_tablet_manager.cc:1398] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:08.876125  3942 tablet_bootstrap.cc:654] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:08.879732  3942 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:08.879931  3942 ts_tablet_manager.cc:1367] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.001s
I20240503 02:15:08.880875  3942 raft_consensus.cc:348] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.881048  3942 raft_consensus.cc:374] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:08.881108  3942 raft_consensus.cc:729] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:08.881397  3942 consensus_queue.cc:260] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:08.881884  3942 ts_tablet_manager.cc:1398] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:15:08.890321  3781 raft_consensus.cc:482] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:08.890548  3781 raft_consensus.cc:504] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.891250  3781 leader_election.cc:288] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:08.891855  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadcb49d12664e6fb5a5ad8dbe47f342" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:08.892113  2541 raft_consensus.cc:2454] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:08.892529  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadcb49d12664e6fb5a5ad8dbe47f342" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:08.892570  2346 leader_election.cc:302] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:08.892762  2672 raft_consensus.cc:2454] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 0.
I20240503 02:15:08.892863  3781 raft_consensus.cc:2781] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:08.892966  3781 raft_consensus.cc:482] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:08.893083  3781 raft_consensus.cc:3037] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:08.895131  3781 raft_consensus.cc:504] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.895803  3781 leader_election.cc:288] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:08.896500  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadcb49d12664e6fb5a5ad8dbe47f342" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:08.896584  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eadcb49d12664e6fb5a5ad8dbe47f342" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:08.896831  2672 raft_consensus.cc:3037] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:08.896889  2541 raft_consensus.cc:3037] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:08.899086  2541 raft_consensus.cc:2454] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:08.899044  2672 raft_consensus.cc:2454] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 1.
I20240503 02:15:08.899739  2346 leader_election.cc:302] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:08.900156  3781 raft_consensus.cc:2781] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:08.900372  3781 raft_consensus.cc:686] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:08.900645  3781 consensus_queue.cc:237] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:08.904836  2132 catalog_manager.cc:5484] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:09.127966  3890 raft_consensus.cc:482] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:09.128271  3890 raft_consensus.cc:504] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:09.128957  3890 leader_election.cc:288] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:09.129624  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dbf0fd54b14c2fa21d3e62104c9b87" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:09.129626  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dbf0fd54b14c2fa21d3e62104c9b87" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:09.129892  2410 raft_consensus.cc:2454] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:09.129889  2281 raft_consensus.cc:2454] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:09.130378  2478 leader_election.cc:302] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:09.130740  3890 raft_consensus.cc:2781] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:09.130836  3890 raft_consensus.cc:482] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:09.130955  3890 raft_consensus.cc:3037] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.133137  3890 raft_consensus.cc:504] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:09.133895  3890 leader_election.cc:288] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:09.134423  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dbf0fd54b14c2fa21d3e62104c9b87" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:09.134634  2410 raft_consensus.cc:3037] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.134654  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dbf0fd54b14c2fa21d3e62104c9b87" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:09.134976  2281 raft_consensus.cc:3037] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.136726  2410 raft_consensus.cc:2454] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:09.137153  2281 raft_consensus.cc:2454] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:09.137373  2475 leader_election.cc:302] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:09.137825  3890 raft_consensus.cc:2781] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:09.138077  3890 raft_consensus.cc:686] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:09.138365  3890 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:15:09.141918  2132 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:15:09.148103  3754 raft_consensus.cc:482] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:09.148330  3754 raft_consensus.cc:504] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:09.148941  3754 leader_election.cc:288] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:09.149674  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a088d230d0462980648628eae548d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:09.149674  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a088d230d0462980648628eae548d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:09.149937  2672 raft_consensus.cc:2454] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:09.149937  2410 raft_consensus.cc:2454] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 0.
I20240503 02:15:09.150522  2216 leader_election.cc:302] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:09.150977  3754 raft_consensus.cc:2781] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:09.151118  3754 raft_consensus.cc:482] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:09.151263  3754 raft_consensus.cc:3037] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.153877  3754 raft_consensus.cc:504] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:09.154865  3754 leader_election.cc:288] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:09.155364  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a088d230d0462980648628eae548d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:09.155609  2672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a088d230d0462980648628eae548d2" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:09.155737  2410 raft_consensus.cc:3037] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.155903  2672 raft_consensus.cc:3037] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:09.159198  2410 raft_consensus.cc:2454] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:09.159198  2672 raft_consensus.cc:2454] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 1.
I20240503 02:15:09.159899  2214 leader_election.cc:302] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:09.160329  3754 raft_consensus.cc:2781] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:09.160624  3754 raft_consensus.cc:686] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:09.160970  3754 consensus_queue.cc:237] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:09.164520  2132 catalog_manager.cc:5484] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:09.297294  3881 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:09.307585  3881 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:09.594658  3718 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:09.603443  3966 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:09.666329  3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:09.666743  3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:09.675633  3959 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 02:15:09.732432  3754 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:09.742053  3754 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 263.848 ms
  time per row: 0.131924 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      6       1788 libcrypto.so.1.1
-----------------------------------------------------

I20240503 02:15:11.084897  2137 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56732:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:15:11.097469  2521 tablet_service.cc:1467] Processing CreateTablet for tablet 2de1d29dbea945eba58f8eadb019d766 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.097482  2259 tablet_service.cc:1467] Processing CreateTablet for tablet 2de1d29dbea945eba58f8eadb019d766 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.097950  2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de1d29dbea945eba58f8eadb019d766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.098009  2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de1d29dbea945eba58f8eadb019d766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.098323  2652 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61eb7d9e6041b6b99d5d5328943140 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.098771  2652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61eb7d9e6041b6b99d5d5328943140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.101915  2651 tablet_service.cc:1467] Processing CreateTablet for tablet 2de1d29dbea945eba58f8eadb019d766 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.102381  2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de1d29dbea945eba58f8eadb019d766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.105681  2522 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61eb7d9e6041b6b99d5d5328943140 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.106159  2522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61eb7d9e6041b6b99d5d5328943140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.107460  2520 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.107861  2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.110579  2261 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.111085  2261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.111843  4027 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:11.112003  4029 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:11.114064  4027 tablet_bootstrap.cc:654] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.114729  4028 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:11.114845  4029 tablet_bootstrap.cc:654] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.114905  2260 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61eb7d9e6041b6b99d5d5328943140 (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.115358  2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61eb7d9e6041b6b99d5d5328943140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.116681  4028 tablet_bootstrap.cc:654] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.119160  2391 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece (DEFAULT_TABLE table=rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:15:11.119812  2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:11.122598  4027 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:11.123157  4027 ts_tablet_manager.cc:1367] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240503 02:15:11.124027  4027 raft_consensus.cc:348] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.124238  4027 raft_consensus.cc:374] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.124320  4027 raft_consensus.cc:729] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.124644  4027 consensus_queue.cc:260] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.125308  4027 ts_tablet_manager.cc:1398] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:11.125412  4028 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:11.125584  4028 ts_tablet_manager.cc:1367] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240503 02:15:11.125627  4027 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:11.127413  4027 tablet_bootstrap.cc:654] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.127882  4028 raft_consensus.cc:348] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.128203  4028 raft_consensus.cc:374] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.128363  4028 raft_consensus.cc:729] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.128680  4028 consensus_queue.cc:260] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.134392  4029 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:11.134732  4029 ts_tablet_manager.cc:1367] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.004s
I20240503 02:15:11.135890  4029 raft_consensus.cc:348] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.136723  4029 raft_consensus.cc:374] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.137100  4029 raft_consensus.cc:729] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.137674  4029 consensus_queue.cc:260] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.143430  4028 ts_tablet_manager.cc:1398] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:15:11.143827  4028 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:11.145555  4028 tablet_bootstrap.cc:654] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.146051  4029 ts_tablet_manager.cc:1398] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.001s
I20240503 02:15:11.148612  4029 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:15:11.154350  4034 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:15:11.157181  4029 tablet_bootstrap.cc:654] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.158396  4034 tablet_bootstrap.cc:654] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.159787  4027 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:11.159993  4027 ts_tablet_manager.cc:1367] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.021s
I20240503 02:15:11.160884  4027 raft_consensus.cc:348] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.161140  4027 raft_consensus.cc:374] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.161227  4027 raft_consensus.cc:729] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.161456  4029 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83: No bootstrap required, opened a new log
I20240503 02:15:11.161648  4029 ts_tablet_manager.cc:1367] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.002s
I20240503 02:15:11.162153  4028 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:11.162019  4027 consensus_queue.cc:260] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.162490  4029 raft_consensus.cc:348] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.162698  4034 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234: No bootstrap required, opened a new log
I20240503 02:15:11.162720  4027 ts_tablet_manager.cc:1398] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240503 02:15:11.162807  4029 raft_consensus.cc:374] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.162889  4034 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240503 02:15:11.162992  4029 raft_consensus.cc:729] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.163167  4027 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:15:11.163332  4028 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.007s
I20240503 02:15:11.163368  4029 consensus_queue.cc:260] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.164104  4028 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.164435  4028 raft_consensus.cc:374] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.164574  4028 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.164623  3966 raft_consensus.cc:482] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:11.164777  3966 raft_consensus.cc:504] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.164896  4028 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.165234  4029 ts_tablet_manager.cc:1398] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:11.165658  3966 leader_election.cc:288] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:11.166347  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de1d29dbea945eba58f8eadb019d766" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:11.166605  2281 raft_consensus.cc:2454] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:11.166584  4028 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:11.167447  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de1d29dbea945eba58f8eadb019d766" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83" is_pre_election: true
I20240503 02:15:11.167577  2478 leader_election.cc:302] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:11.167748  2673 raft_consensus.cc:2454] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:11.167904  3966 raft_consensus.cc:2781] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:11.168001  3966 raft_consensus.cc:482] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:11.168102  3966 raft_consensus.cc:3037] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.168255  4028 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:15:11.168774  4034 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.168962  4034 raft_consensus.cc:374] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.169072  4034 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.169310  4034 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.170410  4034 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.002s
I20240503 02:15:11.170924  3966 raft_consensus.cc:504] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.172211  4028 tablet_bootstrap.cc:654] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.172976  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de1d29dbea945eba58f8eadb019d766" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:11.173189  2281 raft_consensus.cc:3037] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.177525  2281 raft_consensus.cc:2454] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:11.177675  3966 leader_election.cc:288] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:11.178207  2673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de1d29dbea945eba58f8eadb019d766" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:11.178390  2478 leader_election.cc:302] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:11.178490  2673 raft_consensus.cc:3037] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.178763  4027 tablet_bootstrap.cc:654] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Neither blocks nor log segments found. Creating new log.
I20240503 02:15:11.179025  3966 raft_consensus.cc:2781] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:11.182426  2673 raft_consensus.cc:2454] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:11.186064  4028 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850: No bootstrap required, opened a new log
I20240503 02:15:11.186491  4028 ts_tablet_manager.cc:1367] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.009s
I20240503 02:15:11.188191  4028 raft_consensus.cc:348] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.188680  4028 raft_consensus.cc:374] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.188858  4028 raft_consensus.cc:729] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.189209  4028 consensus_queue.cc:260] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.189525  3800 raft_consensus.cc:482] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:11.189697  3800 raft_consensus.cc:504] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.190063  4028 ts_tablet_manager.cc:1398] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:15:11.190508  3800 leader_election.cc:288] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:11.191176  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61eb7d9e6041b6b99d5d5328943140" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:11.191442  2281 raft_consensus.cc:2454] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:11.191695  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61eb7d9e6041b6b99d5d5328943140" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852" is_pre_election: true
I20240503 02:15:11.191936  2542 raft_consensus.cc:2454] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 0.
I20240503 02:15:11.191972  2609 leader_election.cc:302] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:11.192130  4027 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: No bootstrap required, opened a new log
I20240503 02:15:11.192265  3800 raft_consensus.cc:2781] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:11.192284  4027 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.029s	user 0.001s	sys 0.010s
I20240503 02:15:11.192382  3800 raft_consensus.cc:482] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:11.192479  3800 raft_consensus.cc:3037] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.193097  4027 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.193269  4027 raft_consensus.cc:374] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:15:11.193408  4027 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:15:11.193640  4027 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.194197  4027 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:11.195279  3800 raft_consensus.cc:504] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.197088  3966 raft_consensus.cc:686] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:11.197439  3966 consensus_queue.cc:237] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:15:11.197697  3800 leader_election.cc:288] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:11.197777  2542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61eb7d9e6041b6b99d5d5328943140" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:11.198010  2542 raft_consensus.cc:3037] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.201135  2542 raft_consensus.cc:2454] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:11.201771  2607 leader_election.cc:302] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:15:11.202088  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61eb7d9e6041b6b99d5d5328943140" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:11.201841  2138 catalog_manager.cc:5484] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:15:11.202354  2281 raft_consensus.cc:3037] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.202790  3966 raft_consensus.cc:482] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:15:11.203014  3966 raft_consensus.cc:504] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.203538  4043 raft_consensus.cc:2781] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:11.203761  3966 leader_election.cc:288] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:11.204279  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec063c94fe941ebb1c866b5b8935ece" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850" is_pre_election: true
I20240503 02:15:11.204380  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec063c94fe941ebb1c866b5b8935ece" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234" is_pre_election: true
I20240503 02:15:11.204514  2279 raft_consensus.cc:2454] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:11.204596  2412 raft_consensus.cc:2454] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 0.
I20240503 02:15:11.205157  2475 leader_election.cc:302] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:11.205950  3966 raft_consensus.cc:2781] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:15:11.206123  3966 raft_consensus.cc:482] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:15:11.206216  3966 raft_consensus.cc:3037] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.208302  2281 raft_consensus.cc:2454] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 1.
I20240503 02:15:11.208930  3966 raft_consensus.cc:504] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.210208  3966 leader_election.cc:288] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:11.210574  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec063c94fe941ebb1c866b5b8935ece" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:11.210780  2281 raft_consensus.cc:3037] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.212730  2281 raft_consensus.cc:2454] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:11.213251  2478 leader_election.cc:302] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:11.213467  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec063c94fe941ebb1c866b5b8935ece" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:15:11.213631  3966 raft_consensus.cc:2781] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:15:11.213755  2412 raft_consensus.cc:3037] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:15:11.213721  4043 raft_consensus.cc:686] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:15:11.213900  3966 raft_consensus.cc:686] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:11.214172  3966 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:11.214126  4043 consensus_queue.cc:237] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:15:11.217026  2412 raft_consensus.cc:2454] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 1.
I20240503 02:15:11.217905  2138 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:11.218938  2137 catalog_manager.cc:5484] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:15:11.607935  3966 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:11.614012  3966 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:11.629500  4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:11.629997  4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:11.638947  4047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 02:15:11.704723  3966 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:11.717710  4039 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:11.718361  4044 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:15:11.763942  4044 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:15:11.928246  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 59371.6us

INSERT report
    rows total: 2000
    time total: 292.86 ms
  time per row: 0.14643 ms
I20240503 02:15:12.099807 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:39741
--local_ip_for_outbound_sockets=127.20.218.69
--tserver_master_addrs=127.20.218.126:35007
--webserver_port=35331
--webserver_interface=127.20.218.69
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:15:12.516369  4114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:12.516709  4114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:15:12.516804  4114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:12.524729  4114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:15:12.524889  4114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:15:12.525007  4114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:15:12.535028  4114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:39741
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=35331
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:15:12.536535  4114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:15:12.538544  4114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:15:12.552624  4120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:12.552742  4122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:12.552711  4119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:15:12.553673  4114 server_base.cc:1034] running on GCE node
I20240503 02:15:12.554544  4114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:15:12.555935  4114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:15:12.557195  4114 hybrid_clock.cc:648] HybridClock initialized: now 1714702512557141 us; error 49 us; skew 500 ppm
I20240503 02:15:12.557636  4114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:15:12.561604  4114 webserver.cc:460] Webserver started at http://127.20.218.69:35331/ using document root <none> and password file <none>
I20240503 02:15:12.562709  4114 fs_manager.cc:359] Metadata directory not provided
I20240503 02:15:12.562860  4114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:15:12.569069  4114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.001s
I20240503 02:15:12.572851  4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:12.574676  4114 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:15:12.574894  4114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1d36807868fe4a82bca42c25831ce8a2"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:15:12.575821  4114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:15:12.594245  4114 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:15:12.595309  4114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:15:12.595705  4114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:15:12.596850  4114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:15:12.598567  4114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:15:12.598645  4114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:12.598789  4114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:15:12.598845  4114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:12.633901  4114 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:39741
I20240503 02:15:12.633922  4240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:39741 every 8 connection(s)
I20240503 02:15:12.635676  4114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:15:12.643484 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 4114
I20240503 02:15:12.644515 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.70:33893
--local_ip_for_outbound_sockets=127.20.218.70
--tserver_master_addrs=127.20.218.126:35007
--webserver_port=40999
--webserver_interface=127.20.218.70
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:15:12.654613  4241 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:15:12.654956  4241 heartbeater.cc:464] Registering TS with master...
I20240503 02:15:12.655872  4241 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:15:12.658062  2137 ts_manager.cc:194] Re-registered known tserver with Master: 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:15:12.659313  2137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:59421
W20240503 02:15:13.063447  4245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:13.063783  4245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:15:13.064086  4245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:13.073065  4245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:15:13.073261  4245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:15:13.073468  4245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.70
I20240503 02:15:13.085496  4245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.70:33893
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.218.70
--webserver_port=40999
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.70
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:15:13.087375  4245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:15:13.089874  4245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:15:13.105242  4250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:13.105211  4251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:13.107379  4253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:15:13.107836  4245 server_base.cc:1034] running on GCE node
I20240503 02:15:13.108609  4245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:15:13.109704  4245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:15:13.110916  4245 hybrid_clock.cc:648] HybridClock initialized: now 1714702513110862 us; error 49 us; skew 500 ppm
I20240503 02:15:13.111235  4245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:15:13.114427  4245 webserver.cc:460] Webserver started at http://127.20.218.70:40999/ using document root <none> and password file <none>
I20240503 02:15:13.115595  4245 fs_manager.cc:359] Metadata directory not provided
I20240503 02:15:13.115794  4245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:15:13.123266  4245 fs_manager.cc:709] Time spent opening directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:15:13.126904  4259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.129563  4245 fs_manager.cc:725] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:15:13.129771  4245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e699d9f984b4cf2b8b468322759b78c"
format_stamp: "Formatted at 2024-05-03 02:14:52 on dist-test-slave-81lq"
I20240503 02:15:13.130936  4245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:15:13.158859  4245 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:15:13.159852  4245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:15:13.160161  4245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:15:13.161274  4245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:15:13.163069  4245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:15:13.163152  4245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.163254  4245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:15:13.163301  4245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.201696  4245 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.70:33893
I20240503 02:15:13.201762  4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.70:33893 every 8 connection(s)
I20240503 02:15:13.203300  4245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:15:13.208022 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 4245
I20240503 02:15:13.208807 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.71:38285
--local_ip_for_outbound_sockets=127.20.218.71
--tserver_master_addrs=127.20.218.126:35007
--webserver_port=42735
--webserver_interface=127.20.218.71
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:15:13.220768  4372 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:15:13.221179  4372 heartbeater.cc:464] Registering TS with master...
I20240503 02:15:13.222151  4372 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:15:13.223985  2137 ts_manager.cc:194] Re-registered known tserver with Master: 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:15:13.225255  2137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.70:51951
W20240503 02:15:13.603430  4377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:13.603753  4377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:15:13.603883  4377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:13.612185  4377 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:15:13.612329  4377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:15:13.612501  4377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.71
I20240503 02:15:13.622435  4377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.71:38285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.218.71
--webserver_port=42735
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.71
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:15:13.623895  4377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:15:13.625923  4377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:15:13.640146  4384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:13.641525  4385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:13.641914  4387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:15:13.643360  4377 server_base.cc:1034] running on GCE node
I20240503 02:15:13.644068  4377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:15:13.645277  4377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:15:13.646548  4377 hybrid_clock.cc:648] HybridClock initialized: now 1714702513646485 us; error 55 us; skew 500 ppm
I20240503 02:15:13.646915  4377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:15:13.654644  4377 webserver.cc:460] Webserver started at http://127.20.218.71:42735/ using document root <none> and password file <none>
I20240503 02:15:13.655488  4377 fs_manager.cc:359] Metadata directory not provided
I20240503 02:15:13.655637  4377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:15:13.661582  4377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:15:13.663146  4241 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:15:13.665624  4393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.667718  4377 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:15:13.668016  4377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "408af0ab5520465e942cc714bfc6c73b"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:15:13.669157  4377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:15:13.688319  4377 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:15:13.689392  4377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:15:13.689754  4377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:15:13.690941  4377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:15:13.692550  4377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:15:13.692628  4377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.692734  4377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:15:13.692776  4377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:13.732555  4377 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.71:38285
I20240503 02:15:13.732594  4505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.71:38285 every 8 connection(s)
I20240503 02:15:13.734084  4377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:15:13.739602 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 4377
I20240503 02:15:13.740412 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.72:37033
--local_ip_for_outbound_sockets=127.20.218.72
--tserver_master_addrs=127.20.218.126:35007
--webserver_port=42969
--webserver_interface=127.20.218.72
--builtin_ntp_servers=127.20.218.84:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:15:13.750196  4506 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:15:13.750833  4506 heartbeater.cc:464] Registering TS with master...
I20240503 02:15:13.751879  4506 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:15:13.753870  2137 ts_manager.cc:194] Re-registered known tserver with Master: 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:15:13.754841  2137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.71:54443
W20240503 02:15:14.140473  4510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:14.140838  4510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:15:14.140995  4510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:14.149292  4510 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:15:14.149504  4510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:15:14.149678  4510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.72
I20240503 02:15:14.160975  4510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.72:37033
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.218.72
--webserver_port=42969
--tserver_master_addrs=127.20.218.126:35007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.72
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:15:14.162645  4510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:15:14.164887  4510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:15:14.179654  4516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:14.179723  4518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:15:14.179755  4515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:15:14.181304  4510 server_base.cc:1034] running on GCE node
I20240503 02:15:14.182766  4510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:15:14.184154  4510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:15:14.185434  4510 hybrid_clock.cc:648] HybridClock initialized: now 1714702514185363 us; error 74 us; skew 500 ppm
I20240503 02:15:14.185855  4510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:15:14.188452  4510 webserver.cc:460] Webserver started at http://127.20.218.72:42969/ using document root <none> and password file <none>
I20240503 02:15:14.189411  4510 fs_manager.cc:359] Metadata directory not provided
I20240503 02:15:14.189548  4510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:15:14.195752  4510 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:15:14.199481  4524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:14.201455  4510 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:15:14.201665  4510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7fd07044862a481aa387698017467c01"
format_stamp: "Formatted at 2024-05-03 02:14:53 on dist-test-slave-81lq"
I20240503 02:15:14.202839  4510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:15:14.228447  4372 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
I20240503 02:15:14.230290  4510 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:15:14.231438  4510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:15:14.231842  4510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:15:14.241850  4510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:15:14.247182  4510 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:15:14.247313  4510 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:14.247449  4510 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:15:14.247505  4510 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:15:14.322151  4510 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.72:37033
I20240503 02:15:14.322188  4638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.72:37033 every 8 connection(s)
I20240503 02:15:14.323808  4510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:15:14.333271 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 4510
I20240503 02:15:14.342861  4639 heartbeater.cc:347] Connected to a master server at 127.20.218.126:35007
I20240503 02:15:14.343328  4639 heartbeater.cc:464] Registering TS with master...
I20240503 02:15:14.344640  4639 heartbeater.cc:511] Master 127.20.218.126:35007 requested a full tablet report, sending...
I20240503 02:15:14.346716  2137 ts_manager.cc:194] Re-registered known tserver with Master: 7fd07044862a481aa387698017467c01 (127.20.218.72:37033)
I20240503 02:15:14.347986  2137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.72:32999
I20240503 02:15:14.759465  4506 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
W20240503 02:15:15.063951  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 62702.3us
I20240503 02:15:15.352159  4639 heartbeater.cc:503] Master 127.20.218.126:35007 was elected leader, sending a full tablet report...
W20240503 02:15:15.549544  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 132644us
I20240503 02:15:15.917209  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:15.947556  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:15.971295  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:15.994841  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:16.045588  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.061169  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.071197  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:16.084534  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.173779  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:16.177716  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:16.186609  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.201603  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:16.214485  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.258291  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.267251  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.271003  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:16.314443  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:16.338618  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:16.339663  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.351001  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.378516  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:16.404614  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.415555  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:16.455004  2609 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 57135.4us
I20240503 02:15:16.515362  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:16.629170  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:16.657814  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:16.688412  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.744454  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:16.762374  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.842896  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:16.849493  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:16.851091  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.864857  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.869040  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.879339  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.882315  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:16.895591  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:16.924201  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:16.972112  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.041800  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:17.608882  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:17.679821  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:17.685148  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.700531  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:17.711638  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.721765  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:17.735208  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:17.768596  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 53222.9us
I20240503 02:15:17.783730  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:17.995436  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:18.055150  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:18.085505  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.101145  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.122332  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:18.142313  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 77027us
I20240503 02:15:18.145507  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:18.162547  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.219081  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:18.412696  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.422257  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:18.453303  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.453462  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:15:18.457792  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 76461.3us
W20240503 02:15:18.491487  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 60758.4us
I20240503 02:15:18.509738  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.515682  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:18.550776  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.551084  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:18.583415  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:18.593556  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:18.619305  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.700222  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:18.716827  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:18.719719  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.722116  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.748787  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.845742  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:18.872498  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:18.880982  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.884565  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:18.921809  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:18.962616  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.007174  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.009263  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:19.326323  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:19.328107  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.370060  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:19.383984  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:19.384946  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.408174  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.451772  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:19.452752  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.886216  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:19.926640  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:19.963393  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:19.968240  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.019254  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:20.050256  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.111636  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.134778  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:20.177485  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:20.180205  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:20.186347  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:20.186779  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.224650  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:20.232059  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.242308  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:20.251782  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.283015  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.297165  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:20.312767  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.319475  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:20.372190  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.380020  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.423449  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:20.472944  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.578263  2673 consensus_queue.cc:237] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:20.614012  2542 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:20.614816  5337 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:20.620210  2281 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:20.621498  5337 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:20.625674  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:20.689356  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.725159  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:15:20.728237  2609 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 190ebb1335d64b279a41fa38feff22b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:20.741029  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.762832  2542 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:20.770329  2136 catalog_manager.cc:5484] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:20.789788  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:20.761165  5410 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:20.821007  2279 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:20.850382  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:20.882889  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:20.899403  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 58309.6us
I20240503 02:15:20.910774  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:21.045799  2672 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:21.073249  2542 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.074471  5410 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:21.077708  2606 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eeb69e5eb3c459ca749aa37672ec057. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.083081  2412 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.084199  5410 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.098959  5410 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.102789  2410 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.105854  2542 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.107846  2139 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.155179  5448 ts_tablet_manager.cc:902] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:21.182104  5448 tablet_copy_client.cc:308] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:21.244163  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 190ebb1335d64b279a41fa38feff22b4 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:38627)
I20240503 02:15:21.244534  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 190ebb1335d64b279a41fa38feff22b4 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:38627: session id = 408af0ab5520465e942cc714bfc6c73b-190ebb1335d64b279a41fa38feff22b4
I20240503 02:15:21.253137  2693 tablet_copy_source_session.cc:205] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:21.265442  5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190ebb1335d64b279a41fa38feff22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:21.282720  2672 consensus_queue.cc:237] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:21.288224  2279 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:21.291312  2609 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 190ebb1335d64b279a41fa38feff22b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.292502  2542 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:21.294162  5337 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:21.296922  5337 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:21.300631  5337 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
W20240503 02:15:21.302455  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 190ebb1335d64b279a41fa38feff22b4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.306005  2279 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.321012  2140 catalog_manager.cc:5484] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.357961  2542 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.398869  5448 tablet_copy_client.cc:783] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:21.399446  5448 tablet_copy_client.cc:647] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:21.403667  5448 tablet_copy_client.cc:523] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:21.418102  5448 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:21.431057  2279 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:21.455802  5448 log.cc:824] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:21.464486  2542 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.465476  5312 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.513428  2672 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.517069  5312 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.555646  2672 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.564322  2139 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.552697  5536 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
W20240503 02:15:21.568830  2216 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2351807a2ed47419f8d674fbeb9d0ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.634709  2410 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:21.635809  5552 ts_tablet_manager.cc:902] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:21.638250  2542 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:21.696888  5448 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:21.697793  5448 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:21.698513  5448 ts_tablet_manager.cc:1367] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.285s	user 0.110s	sys 0.072s
I20240503 02:15:21.708225  5448 raft_consensus.cc:348] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:21.709779  5563 ts_tablet_manager.cc:902] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:21.716342  5448 raft_consensus.cc:729] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:21.717607  5448 consensus_queue.cc:260] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:21.730525  2279 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.731487  5136 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.733819  5448 ts_tablet_manager.cc:1398] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.006s
I20240503 02:15:21.735944  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-190ebb1335d64b279a41fa38feff22b4 received from {username='slave'} at 127.20.218.71:38627
I20240503 02:15:21.736152  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-190ebb1335d64b279a41fa38feff22b4 on tablet 190ebb1335d64b279a41fa38feff22b4 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:21.757766  2672 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.759341  5136 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:21.772203  5556 log.cc:924] Time spent T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20240503 02:15:21.773743  2348 consensus_peers.cc:485] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0137e646f1bd452b950d870107afa060. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.782117  5136 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.783246  2410 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:21.785463  2673 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.793695  2139 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.817755  2279 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:15:21.822989  2348 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8870feaf47a64a929dd2e584ece47218. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.824644  2542 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.825752  5545 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.833612  2281 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.837410  5577 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.843233  2412 consensus_queue.cc:237] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:21.860343  2541 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.861513  5578 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.865928  2672 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:21.867064  5578 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:21.870275  5563 tablet_copy_client.cc:308] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:21.904353  5600 ts_tablet_manager.cc:902] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:21.909166  2540 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.917795  5578 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.930596  2139 catalog_manager.cc:5484] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.931150  2672 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:21.941812  5136 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.945024  5552 tablet_copy_client.cc:308] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:21.948405  2281 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.949529  2140 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:15:21.965678  2345 consensus_peers.cc:485] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f084f1783d4d477e9c3791984e681f4c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:21.965389  2542 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:21.973587  5600 tablet_copy_client.cc:308] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:21.978667  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 190ebb1335d64b279a41fa38feff22b4 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:42451)
I20240503 02:15:21.978880  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 190ebb1335d64b279a41fa38feff22b4 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:42451: session id = 7fd07044862a481aa387698017467c01-190ebb1335d64b279a41fa38feff22b4
I20240503 02:15:21.995779  2693 tablet_copy_source_session.cc:205] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.000937  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:52945)
I20240503 02:15:22.001405  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:52945: session id = 9e699d9f984b4cf2b8b468322759b78c-1eeb69e5eb3c459ca749aa37672ec057
I20240503 02:15:22.006383  5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190ebb1335d64b279a41fa38feff22b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.007124  2693 tablet_copy_source_session.cc:205] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.020464  5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.061396  5552 tablet_copy_client.cc:783] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.061894  5552 tablet_copy_client.cc:647] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.072443  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet d2351807a2ed47419f8d674fbeb9d0ec from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:49069)
I20240503 02:15:22.072733  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet d2351807a2ed47419f8d674fbeb9d0ec from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:49069: session id = 7fd07044862a481aa387698017467c01-d2351807a2ed47419f8d674fbeb9d0ec
I20240503 02:15:22.073405  5552 tablet_copy_client.cc:523] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.078300  2301 tablet_copy_source_session.cc:205] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.082123  5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.085311  5563 tablet_copy_client.cc:783] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.085734  5563 tablet_copy_client.cc:647] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.101090  5563 tablet_copy_client.cc:523] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.108034  5563 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:22.135159  5448 ts_tablet_manager.cc:902] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:22.135290  5600 tablet_copy_client.cc:783] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.135484  5563 log.cc:824] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:22.135591  5600 tablet_copy_client.cc:647] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.157318  5552 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:22.163552  4461 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.169837  5448 tablet_copy_client.cc:308] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:22.176616  2672 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:22.187026  5600 tablet_copy_client.cc:523] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.192188  5600 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:22.196025  2412 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:22.196903  5498 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:22.208101  2281 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:22.208765  5498 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:22.221091  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 0137e646f1bd452b950d870107afa060 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56697)
I20240503 02:15:22.221500  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 0137e646f1bd452b950d870107afa060 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56697: session id = 408af0ab5520465e942cc714bfc6c73b-0137e646f1bd452b950d870107afa060
I20240503 02:15:22.226898  2432 tablet_copy_source_session.cc:205] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.229575  5448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0137e646f1bd452b950d870107afa060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.237617  5448 tablet_copy_client.cc:783] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.238327  5448 tablet_copy_client.cc:647] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.241209  5448 tablet_copy_client.cc:523] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.245432  5448 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
W20240503 02:15:22.248143  2608 consensus_peers.cc:485] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a3dcc2de594c8982477a0229f71191. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.255573  5563 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:22.257941  5563 tablet_bootstrap.cc:492] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:22.258607  5563 ts_tablet_manager.cc:1367] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.152s	user 0.077s	sys 0.007s
I20240503 02:15:22.298195  5664 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.299129  5552 log.cc:824] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:22.300769  2410 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.308687  2139 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.334686  5563 raft_consensus.cc:348] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:22.335269  5563 raft_consensus.cc:729] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:22.337579  2281 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.343156  5563 consensus_queue.cc:260] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:22.374392  5681 ts_tablet_manager.cc:902] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:22.400451  5600 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:22.403177  5681 tablet_copy_client.cc:308] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:22.413022  5600 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:22.413297  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet f084f1783d4d477e9c3791984e681f4c from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:60421)
I20240503 02:15:22.413513  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet f084f1783d4d477e9c3791984e681f4c from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:60421: session id = 9e699d9f984b4cf2b8b468322759b78c-f084f1783d4d477e9c3791984e681f4c
I20240503 02:15:22.416167  5600 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.224s	user 0.053s	sys 0.012s
I20240503 02:15:22.416638  2432 tablet_copy_source_session.cc:205] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.418218  5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f1783d4d477e9c3791984e681f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.429908  5695 ts_tablet_manager.cc:902] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:22.425619  5563 ts_tablet_manager.cc:1398] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.148s	user 0.021s	sys 0.008s
I20240503 02:15:22.425304  5600 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:22.431491  5600 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:22.433027  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 7fd07044862a481aa387698017467c01-190ebb1335d64b279a41fa38feff22b4 received from {username='slave'} at 127.20.218.72:42451
I20240503 02:15:22.433198  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 7fd07044862a481aa387698017467c01-190ebb1335d64b279a41fa38feff22b4 on tablet 190ebb1335d64b279a41fa38feff22b4 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:22.433805  5600 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:22.435024  5600 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.019s	user 0.002s	sys 0.000s
I20240503 02:15:22.435660  2672 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:22.437315  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 7fd07044862a481aa387698017467c01-d2351807a2ed47419f8d674fbeb9d0ec received from {username='slave'} at 127.20.218.72:49069
I20240503 02:15:22.437558  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 7fd07044862a481aa387698017467c01-d2351807a2ed47419f8d674fbeb9d0ec on tablet d2351807a2ed47419f8d674fbeb9d0ec with peer 7fd07044862a481aa387698017467c01
W20240503 02:15:22.440446  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eeb69e5eb3c459ca749aa37672ec057. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.442139  2410 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:22.442880  2606 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.443121  5664 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:22.446067  2541 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:22.446805  5664 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:22.449352  5681 tablet_copy_client.cc:783] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.453111  5681 tablet_copy_client.cc:647] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.457824  5681 tablet_copy_client.cc:523] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.463899  5681 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:22.485675  5664 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.489071  2412 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.492950  2541 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.499468  4593 raft_consensus.cc:1203] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:22.500020  2139 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.518082  2281 consensus_queue.cc:237] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.526997  5552 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:22.527640  5552 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:22.528190  5552 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.371s	user 0.069s	sys 0.006s
I20240503 02:15:22.528657  5448 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:22.529124  5448 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:22.537137  5552 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.537905  5552 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:22.538910  5552 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.543056  2672 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:22.543918  5312 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:22.535532  5448 ts_tablet_manager.cc:1367] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.290s	user 0.058s	sys 0.009s
I20240503 02:15:22.553614  2541 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:22.554606  5312 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:22.549966  5695 tablet_copy_client.cc:308] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:22.553292  5448 raft_consensus.cc:348] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:22.560772  5448 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:22.561586  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 8870feaf47a64a929dd2e584ece47218 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56697)
I20240503 02:15:22.561797  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 8870feaf47a64a929dd2e584ece47218 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56697: session id = 408af0ab5520465e942cc714bfc6c73b-8870feaf47a64a929dd2e584ece47218
I20240503 02:15:22.561193  5448 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:22.564622  5448 ts_tablet_manager.cc:1398] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.013s	user 0.000s	sys 0.003s
I20240503 02:15:22.566080  2432 tablet_copy_source_session.cc:205] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.567780  2431 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-0137e646f1bd452b950d870107afa060 received from {username='slave'} at 127.20.218.71:56697
I20240503 02:15:22.567988  2431 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-0137e646f1bd452b950d870107afa060 on tablet 0137e646f1bd452b950d870107afa060 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:22.569306  2281 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:22.572288  5552 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.044s	user 0.020s	sys 0.024s
I20240503 02:15:22.573318  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-1eeb69e5eb3c459ca749aa37672ec057 received from {username='slave'} at 127.20.218.70:52945
I20240503 02:15:22.573526  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-1eeb69e5eb3c459ca749aa37672ec057 on tablet 1eeb69e5eb3c459ca749aa37672ec057 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:22.581028  5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.601850  5729 ts_tablet_manager.cc:902] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:22.608492  5312 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.610376  2542 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.614647  5695 tablet_copy_client.cc:783] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:22.615862  5695 tablet_copy_client.cc:647] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:22.618656  5695 tablet_copy_client.cc:523] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:22.622275  2140 catalog_manager.cc:5484] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.628415  2412 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:22.645987  5548 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:22.648265  5681 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:22.648785  5681 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:22.649153  5681 ts_tablet_manager.cc:1367] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.185s	user 0.054s	sys 0.036s
I20240503 02:15:22.650719  5681 raft_consensus.cc:348] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.651012  5681 raft_consensus.cc:729] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:22.651270  5681 consensus_queue.cc:260] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.651872  5681 ts_tablet_manager.cc:1398] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:15:22.654141  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-f084f1783d4d477e9c3791984e681f4c received from {username='slave'} at 127.20.218.70:60421
I20240503 02:15:22.654325  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-f084f1783d4d477e9c3791984e681f4c on tablet f084f1783d4d477e9c3791984e681f4c with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:22.658176  5729 tablet_copy_client.cc:308] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
W20240503 02:15:22.662601  2214 consensus_peers.cc:485] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 586ebf467f0f427fa62dee6223a39848. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.673738  5695 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:22.681874  2671 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:22.685580  2216 consensus_peers.cc:485] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51224b6355ed41dbb0703b7feb73c223. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.686079  2672 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.687695  2412 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:22.688405  5548 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:22.691073  2345 consensus_peers.cc:485] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0137e646f1bd452b950d870107afa060. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:22.693019  4461 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:22.693821  5577 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:22.697839  2669 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:22.698750  5577 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:22.705193  2281 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:22.706283  5577 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:22.709676  5580 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.710923  5312 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.716179  2668 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.721951  2281 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.727962  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 62a3dcc2de594c8982477a0229f71191 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:44663)
I20240503 02:15:22.728178  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 62a3dcc2de594c8982477a0229f71191 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:44663: session id = 1d36807868fe4a82bca42c25831ce8a2-62a3dcc2de594c8982477a0229f71191
I20240503 02:15:22.728307  2140 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.732012  2693 tablet_copy_source_session.cc:205] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:22.743273  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:22.754310  2412 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:22.757800  4327 raft_consensus.cc:1203] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:22.782038  4593 raft_consensus.cc:1203] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:22.783521  2138 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:15:22.798202  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 66490.8us
I20240503 02:15:22.813601  2668 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.819187  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:22.837071  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:22.844365  4461 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:22.867244  5729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:22.883430  4327 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:22.914773  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:22.950543  5681 ts_tablet_manager.cc:902] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:22.955418  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:22.957731  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:22.995638  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:23.014245  5681 tablet_copy_client.cc:308] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:23.034474  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:23.062680  5729 tablet_copy_client.cc:783] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.063232  5729 tablet_copy_client.cc:647] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.064241  5809 ts_tablet_manager.cc:902] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:23.072898  5729 tablet_copy_client.cc:523] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.099397  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 586ebf467f0f427fa62dee6223a39848 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:34139)
I20240503 02:15:23.099730  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 586ebf467f0f427fa62dee6223a39848 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:34139: session id = 9e699d9f984b4cf2b8b468322759b78c-586ebf467f0f427fa62dee6223a39848
I20240503 02:15:23.103559  2301 tablet_copy_source_session.cc:205] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.105665  5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586ebf467f0f427fa62dee6223a39848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.114018  5681 tablet_copy_client.cc:783] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.114478  5681 tablet_copy_client.cc:647] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.115078  5809 tablet_copy_client.cc:308] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:23.115933  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:42451)
I20240503 02:15:23.116164  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:42451: session id = 7fd07044862a481aa387698017467c01-1eeb69e5eb3c459ca749aa37672ec057
I20240503 02:15:23.119235  2668 consensus_queue.cc:237] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.119979  2693 tablet_copy_source_session.cc:205] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.122330  5809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.131942  5809 tablet_copy_client.cc:783] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.135476  5695 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.136001  5695 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
W20240503 02:15:23.136026  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1949a5c7cc741b19da021eccb7d0dd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:23.136332  5695 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.489s	user 0.050s	sys 0.015s
I20240503 02:15:23.136365  5681 tablet_copy_client.cc:523] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.137348  5695 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:23.137744  5695 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.138116  5695 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:23.139061  5695 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:15:23.139839  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-8870feaf47a64a929dd2e584ece47218 received from {username='slave'} at 127.20.218.71:56697
I20240503 02:15:23.140009  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-8870feaf47a64a929dd2e584ece47218 on tablet 8870feaf47a64a929dd2e584ece47218 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:23.146391  2541 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.148103  2412 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.149406  5671 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.150100  5671 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.155635  2281 consensus_queue.cc:237] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.159081  5729 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:23.132378  5809 tablet_copy_client.cc:647] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.166534  2410 raft_consensus.cc:1261] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.167335  5312 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:23.171097  2216 consensus_peers.cc:485] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0a088d230d0462980648628eae548d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:23.171612  5681 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:23.176154  2668 raft_consensus.cc:1261] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.176224  5809 tablet_copy_client.cc:523] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.177141  5729 log.cc:824] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:15:23.178088  5763 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.184500  5809 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:23.212865  5763 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.219429  5671 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.226087  2410 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.230751  2542 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.238719  4461 raft_consensus.cc:1203] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:23.238772  2140 catalog_manager.cc:5484] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.246336  2140 catalog_manager.cc:5484] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.257012  2668 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.258843  2412 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:23.269943  5820 ts_tablet_manager.cc:902] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:23.277482  5822 ts_tablet_manager.cc:902] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:23.345506  5820 tablet_copy_client.cc:308] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:23.354516  5729 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.355306  5729 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:23.361606  5729 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.203s	user 0.082s	sys 0.002s
I20240503 02:15:23.383549  5809 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.384078  5809 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:23.384357  5809 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.200s	user 0.063s	sys 0.018s
I20240503 02:15:23.386387  5822 tablet_copy_client.cc:308] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:23.387662  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 0137e646f1bd452b950d870107afa060 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:60421)
I20240503 02:15:23.387921  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 0137e646f1bd452b950d870107afa060 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:60421: session id = 9e699d9f984b4cf2b8b468322759b78c-0137e646f1bd452b950d870107afa060
I20240503 02:15:23.391418  2432 tablet_copy_source_session.cc:205] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.393234  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 51224b6355ed41dbb0703b7feb73c223 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:35505)
I20240503 02:15:23.393050  5809 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:23.393442  5822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0137e646f1bd452b950d870107afa060. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.393446  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 51224b6355ed41dbb0703b7feb73c223 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:35505: session id = 1d36807868fe4a82bca42c25831ce8a2-51224b6355ed41dbb0703b7feb73c223
I20240503 02:15:23.393656  5809 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.394129  5809 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:23.396749  2301 tablet_copy_source_session.cc:205] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.400837  2281 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:23.404042  5820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51224b6355ed41dbb0703b7feb73c223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.390803  5729 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.411669  5729 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.413615  5729 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.424022  2668 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.424686  2412 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.425789  5763 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.427289  5763 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.433076  5820 tablet_copy_client.cc:783] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.435387  5820 tablet_copy_client.cc:647] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.442126  5729 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.079s	user 0.006s	sys 0.006s
I20240503 02:15:23.442770  5820 tablet_copy_client.cc:523] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.444355  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-62a3dcc2de594c8982477a0229f71191 received from {username='slave'} at 127.20.218.69:44663
I20240503 02:15:23.444523  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-62a3dcc2de594c8982477a0229f71191 on tablet 62a3dcc2de594c8982477a0229f71191 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:23.448522  5820 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:23.451581  2668 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.454713  5809 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.070s	user 0.014s	sys 0.021s
I20240503 02:15:23.456111  2412 consensus_queue.cc:237] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:23.457892  2139 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.450486  5834 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.469933  5822 tablet_copy_client.cc:783] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.470253  5822 tablet_copy_client.cc:647] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
W20240503 02:15:23.471028  2217 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e0b954008c4f30af414da6cb5d59c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:23.472041  2410 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.484865  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 7fd07044862a481aa387698017467c01-1eeb69e5eb3c459ca749aa37672ec057 received from {username='slave'} at 127.20.218.72:42451
I20240503 02:15:23.485100  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 7fd07044862a481aa387698017467c01-1eeb69e5eb3c459ca749aa37672ec057 on tablet 1eeb69e5eb3c459ca749aa37672ec057 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:23.488672  5822 tablet_copy_client.cc:523] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.492745  5822 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:23.498544  2542 raft_consensus.cc:1261] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:23.499306  2348 consensus_peers.cc:485] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6e0d39087944e4ea13a17ca4af478ab. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:23.499539  2279 raft_consensus.cc:1261] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.500242  5580 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.501734  5580 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.533324  5580 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.536810  2281 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.543015  5681 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.544138  5681 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:23.544457  5681 ts_tablet_manager.cc:1367] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.403s	user 0.052s	sys 0.012s
I20240503 02:15:23.547041  5681 raft_consensus.cc:348] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:23.547314  5681 raft_consensus.cc:729] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.547999  5681 consensus_queue.cc:260] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:23.552817  2140 catalog_manager.cc:5484] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.560712  5681 ts_tablet_manager.cc:1398] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.016s	user 0.001s	sys 0.004s
I20240503 02:15:23.567190  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-586ebf467f0f427fa62dee6223a39848 received from {username='slave'} at 127.20.218.70:34139
I20240503 02:15:23.567415  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-586ebf467f0f427fa62dee6223a39848 on tablet 586ebf467f0f427fa62dee6223a39848 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:23.571343  4593 raft_consensus.cc:1203] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:23.575332  5822 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.575846  5822 tablet_bootstrap.cc:492] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:23.576170  5822 ts_tablet_manager.cc:1367] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.084s	user 0.057s	sys 0.000s
I20240503 02:15:23.577090  5822 raft_consensus.cc:348] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:23.577409  5822 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.577777  5822 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:23.578488  5822 ts_tablet_manager.cc:1398] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:15:23.579433  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-0137e646f1bd452b950d870107afa060 received from {username='slave'} at 127.20.218.70:60421
I20240503 02:15:23.579612  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-0137e646f1bd452b950d870107afa060 on tablet 0137e646f1bd452b950d870107afa060 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:23.585592  2541 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:23.603120  5820 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:23.603735  5820 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:23.604138  5820 ts_tablet_manager.cc:1367] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.156s	user 0.051s	sys 0.009s
I20240503 02:15:23.604835  2412 consensus_queue.cc:237] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:23.605460  5820 raft_consensus.cc:348] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.605737  5820 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:23.605989  5820 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:23.607201  5820 ts_tablet_manager.cc:1398] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:23.608947  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-51224b6355ed41dbb0703b7feb73c223 received from {username='slave'} at 127.20.218.69:35505
I20240503 02:15:23.609134  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-51224b6355ed41dbb0703b7feb73c223 on tablet 51224b6355ed41dbb0703b7feb73c223 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:23.609745  2541 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.610878  5904 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:23.611997  2345 consensus_peers.cc:485] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eadcb49d12664e6fb5a5ad8dbe47f342. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:23.621131  2668 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:23.624029  5580 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:23.663892  4327 raft_consensus.cc:1203] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:23.682250  5577 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:23.683262  2541 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:23.687357  5820 ts_tablet_manager.cc:902] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:23.690171  2139 catalog_manager.cc:5484] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:23.694813  5729 ts_tablet_manager.cc:902] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:23.715152  2668 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:23.722040  4196 raft_consensus.cc:1203] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:23.759613  4196 raft_consensus.cc:1203] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:23.809077  5820 tablet_copy_client.cc:308] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:23.810133  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet f0a088d230d0462980648628eae548d2 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:35505)
I20240503 02:15:23.810403  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet f0a088d230d0462980648628eae548d2 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:35505: session id = 1d36807868fe4a82bca42c25831ce8a2-f0a088d230d0462980648628eae548d2
I20240503 02:15:23.814795  2301 tablet_copy_source_session.cc:205] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.816715  5820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a088d230d0462980648628eae548d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.826584  5729 tablet_copy_client.cc:308] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:23.827981  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet a1949a5c7cc741b19da021eccb7d0dd1 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:44663)
I20240503 02:15:23.828203  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet a1949a5c7cc741b19da021eccb7d0dd1 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:44663: session id = 1d36807868fe4a82bca42c25831ce8a2-a1949a5c7cc741b19da021eccb7d0dd1
I20240503 02:15:23.831678  2693 tablet_copy_source_session.cc:205] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:23.834337  5729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1949a5c7cc741b19da021eccb7d0dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:23.849016  5820 tablet_copy_client.cc:783] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.849754  5820 tablet_copy_client.cc:647] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.852545  5820 tablet_copy_client.cc:523] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.859611  5729 tablet_copy_client.cc:783] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:23.860280  5729 tablet_copy_client.cc:647] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:23.863729  5729 tablet_copy_client.cc:523] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:23.857285  5820 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:23.890395  5729 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:23.948666  2542 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:24.007665  2279 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.008599  5764 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.023224  2412 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.024084  5764 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.032589  5820 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.033171  5820 tablet_bootstrap.cc:492] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:24.033501  5820 ts_tablet_manager.cc:1367] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.176s	user 0.062s	sys 0.011s
I20240503 02:15:24.034749  5820 raft_consensus.cc:348] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.035065  5820 raft_consensus.cc:729] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.035364  5820 consensus_queue.cc:260] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.036083  5820 ts_tablet_manager.cc:1398] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:24.036619  5822 ts_tablet_manager.cc:902] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:24.036906  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-f0a088d230d0462980648628eae548d2 received from {username='slave'} at 127.20.218.69:35505
I20240503 02:15:24.037068  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-f0a088d230d0462980648628eae548d2 on tablet f0a088d230d0462980648628eae548d2 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:24.040727  5822 tablet_copy_client.cc:308] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:24.041260  5729 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.042023  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet eadcb49d12664e6fb5a5ad8dbe47f342 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:60421)
I20240503 02:15:24.042212  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet eadcb49d12664e6fb5a5ad8dbe47f342 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:60421: session id = 9e699d9f984b4cf2b8b468322759b78c-eadcb49d12664e6fb5a5ad8dbe47f342
I20240503 02:15:24.042436  5729 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:24.042738  5729 ts_tablet_manager.cc:1367] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.152s	user 0.053s	sys 0.019s
I20240503 02:15:24.043558  5729 raft_consensus.cc:348] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.043818  5729 raft_consensus.cc:729] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.044051  5729 consensus_queue.cc:260] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.044986  5729 ts_tablet_manager.cc:1398] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:15:24.045850  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-a1949a5c7cc741b19da021eccb7d0dd1 received from {username='slave'} at 127.20.218.69:44663
I20240503 02:15:24.045981  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-a1949a5c7cc741b19da021eccb7d0dd1 on tablet a1949a5c7cc741b19da021eccb7d0dd1 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:24.047444  2432 tablet_copy_source_session.cc:205] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.048880  6006 ts_tablet_manager.cc:902] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:24.048981  5822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadcb49d12664e6fb5a5ad8dbe47f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.056088  5822 tablet_copy_client.cc:783] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.056377  5822 tablet_copy_client.cc:647] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.047641  5994 ts_tablet_manager.cc:902] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:24.061754  6006 tablet_copy_client.cc:308] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:24.062914  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet a6e0d39087944e4ea13a17ca4af478ab from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56697)
I20240503 02:15:24.063158  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet a6e0d39087944e4ea13a17ca4af478ab from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56697: session id = 408af0ab5520465e942cc714bfc6c73b-a6e0d39087944e4ea13a17ca4af478ab
I20240503 02:15:24.064181  5764 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.066087  2281 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.066501  2432 tablet_copy_source_session.cc:205] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.068591  6006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6e0d39087944e4ea13a17ca4af478ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.073201  2280 consensus_queue.cc:237] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.078521  6006 tablet_copy_client.cc:783] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.078922  6006 tablet_copy_client.cc:647] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.086692  2412 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.093137  2133 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.099382  5822 tablet_copy_client.cc:523] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.100992  6006 tablet_copy_client.cc:523] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.103899  5822 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:24.106185  4327 raft_consensus.cc:1203] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 02:15:24.109867  2477 consensus_peers.cc:485] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ec063c94fe941ebb1c866b5b8935ece. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:24.111202  6006 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:24.115350  2412 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.115922  2668 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:24.117084  2216 consensus_peers.cc:485] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 734027e195414b15811ac9d32b132db8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:24.117039  2279 consensus_queue.cc:237] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:24.117700  5548 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.122620  6008 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.131464  2542 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:24.131870  2217 consensus_peers.cc:485] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75ba33132e784f408f3e26882aea3d15. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:24.132098  6008 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.134878  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:24.136593  2410 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.140380  6008 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.144675  6028 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:24.144973  5548 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:24.152418  2410 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:24.159790  2542 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:24.164357  2133 catalog_manager.cc:5484] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:24.167474  2410 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:24.169731  2139 catalog_manager.cc:5484] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:24.177435  2668 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:24.183941  5994 tablet_copy_client.cc:308] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:24.210170  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:24.291963  4196 raft_consensus.cc:1203] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:24.304821  2668 consensus_queue.cc:237] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:24.306912  4196 raft_consensus.cc:1203] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:24.316015  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:24.318452  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56489)
I20240503 02:15:24.318646  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56489: session id = 408af0ab5520465e942cc714bfc6c73b-02e0b954008c4f30af414da6cb5d59c2
I20240503 02:15:24.322047  2301 tablet_copy_source_session.cc:205] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.323840  2542 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.324160  5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.324630  5664 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:24.333595  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e61eb7d9e6041b6b99d5d5328943140. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:24.336331  2279 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.337347  5664 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.346809  5994 tablet_copy_client.cc:783] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.347138  5994 tablet_copy_client.cc:647] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.349972  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:24.361974  5994 tablet_copy_client.cc:523] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.366474  5994 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:24.380678  6006 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.381178  6006 tablet_bootstrap.cc:492] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:24.381543  6006 ts_tablet_manager.cc:1367] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.270s	user 0.075s	sys 0.030s
I20240503 02:15:24.382520  6006 raft_consensus.cc:348] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:24.382795  6006 raft_consensus.cc:729] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.383073  6006 consensus_queue.cc:260] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:24.396908  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:24.422556  5671 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.430881  2541 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.438272  2139 catalog_manager.cc:5484] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.465620  6006 ts_tablet_manager.cc:1398] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.084s	user 0.030s	sys 0.050s
I20240503 02:15:24.466460  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-a6e0d39087944e4ea13a17ca4af478ab received from {username='slave'} at 127.20.218.71:56697
I20240503 02:15:24.466609  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-a6e0d39087944e4ea13a17ca4af478ab on tablet a6e0d39087944e4ea13a17ca4af478ab with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:24.481794  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:24.483692  5822 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.484148  5822 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:24.484411  5822 ts_tablet_manager.cc:1367] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.381s	user 0.057s	sys 0.009s
I20240503 02:15:24.493850  6072 ts_tablet_manager.cc:902] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
W20240503 02:15:24.502538  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 66483us
I20240503 02:15:24.510336  2280 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.485284  5822 raft_consensus.cc:348] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:24.521703  5822 raft_consensus.cc:729] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.522033  5822 consensus_queue.cc:260] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:24.522639  5822 ts_tablet_manager.cc:1398] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.038s	user 0.003s	sys 0.000s
I20240503 02:15:24.524020  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-eadcb49d12664e6fb5a5ad8dbe47f342 received from {username='slave'} at 127.20.218.70:60421
I20240503 02:15:24.524197  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-eadcb49d12664e6fb5a5ad8dbe47f342 on tablet eadcb49d12664e6fb5a5ad8dbe47f342 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:24.528911  4461 raft_consensus.cc:1203] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:24.543707  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:24.550660  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:24.574585  6072 tablet_copy_client.cc:308] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:24.675165  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:24.675534  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:15:24.676507  5994 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.677110  5994 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:24.677644  5994 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.311s	user 0.061s	sys 0.000s
I20240503 02:15:24.678609  5994 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:24.678948  5994 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.679317  5994 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:24.680102  5994 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:15:24.681246  2562 tablet_copy_source_session.cc:205] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.681278  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-02e0b954008c4f30af414da6cb5d59c2 received from {username='slave'} at 127.20.218.71:56489
I20240503 02:15:24.681463  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-02e0b954008c4f30af414da6cb5d59c2 on tablet 02e0b954008c4f30af414da6cb5d59c2 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:24.684036  6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.695200  6072 tablet_copy_client.cc:783] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.695695  6072 tablet_copy_client.cc:647] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.699172  6118 ts_tablet_manager.cc:902] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:24.703297  5994 ts_tablet_manager.cc:902] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:24.704450  6072 tablet_copy_client.cc:523] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.713497  6118 tablet_copy_client.cc:308] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:24.714406  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 734027e195414b15811ac9d32b132db8 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:35505)
I20240503 02:15:24.714711  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 734027e195414b15811ac9d32b132db8 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:35505: session id = 1d36807868fe4a82bca42c25831ce8a2-734027e195414b15811ac9d32b132db8
I20240503 02:15:24.719192  2301 tablet_copy_source_session.cc:205] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.720397  6072 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:24.722124  6118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734027e195414b15811ac9d32b132db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.731189  6118 tablet_copy_client.cc:783] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.731642  6118 tablet_copy_client.cc:647] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.733994  6118 tablet_copy_client.cc:523] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.743886  2412 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:24.755622  6118 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:24.778975  2542 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.779793  6051 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.806835  5994 tablet_copy_client.cc:308] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:24.807718  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 75ba33132e784f408f3e26882aea3d15 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56489)
I20240503 02:15:24.807888  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 75ba33132e784f408f3e26882aea3d15 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56489: session id = 408af0ab5520465e942cc714bfc6c73b-75ba33132e784f408f3e26882aea3d15
I20240503 02:15:24.811174  2301 tablet_copy_source_session.cc:205] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.812937  5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ba33132e784f408f3e26882aea3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.816288  2668 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:24.821147  6128 log.cc:924] Time spent T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20240503 02:15:24.821625  6051 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.833169  5994 tablet_copy_client.cc:783] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.834730  5994 tablet_copy_client.cc:647] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.840983  5994 tablet_copy_client.cc:523] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.848642  5994 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:24.862792  6142 ts_tablet_manager.cc:902] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:24.868382  4327 raft_consensus.cc:1203] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:24.873382  6057 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.880941  2668 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.904917  6142 tablet_copy_client.cc:308] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
W20240503 02:15:24.905722  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b822c691d824b2ba25e967535c9734e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:24.906317  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 0e61eb7d9e6041b6b99d5d5328943140 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:42451)
I20240503 02:15:24.906508  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 0e61eb7d9e6041b6b99d5d5328943140 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:42451: session id = 7fd07044862a481aa387698017467c01-0e61eb7d9e6041b6b99d5d5328943140
I20240503 02:15:24.910535  2139 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.911711  2693 tablet_copy_source_session.cc:205] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:24.914469  6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61eb7d9e6041b6b99d5d5328943140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:24.917810  2542 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:24.961423  6142 tablet_copy_client.cc:783] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:24.961776  6142 tablet_copy_client.cc:647] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:24.962644  2542 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:24.965633  6142 tablet_copy_client.cc:523] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:24.981571  6142 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:24.993544  6118 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:24.993721  2280 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.994052  6118 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:24.994308  2412 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:24.994364  6118 ts_tablet_manager.cc:1367] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.239s	user 0.065s	sys 0.008s
I20240503 02:15:24.995203  6118 raft_consensus.cc:348] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:24.995467  6118 raft_consensus.cc:729] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:24.995743  5917 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:24.995739  6118 consensus_queue.cc:260] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:25.003703  5994 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:25.004196  5994 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:25.004488  5994 ts_tablet_manager.cc:1367] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.156s	user 0.056s	sys 0.002s
I20240503 02:15:25.005498  5994 raft_consensus.cc:348] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:25.005790  5994 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:25.006083  5994 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:25.006876  5994 ts_tablet_manager.cc:1398] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 02:15:25.008100  6118 ts_tablet_manager.cc:1398] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.007s
I20240503 02:15:25.008845  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-75ba33132e784f408f3e26882aea3d15 received from {username='slave'} at 127.20.218.71:56489
I20240503 02:15:25.008988  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-75ba33132e784f408f3e26882aea3d15 on tablet 75ba33132e784f408f3e26882aea3d15 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:25.009756  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-734027e195414b15811ac9d32b132db8 received from {username='slave'} at 127.20.218.69:35505
I20240503 02:15:25.009903  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-734027e195414b15811ac9d32b132db8 on tablet 734027e195414b15811ac9d32b132db8 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:25.027876  2542 consensus_queue.cc:237] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.030500  5917 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:25.037573  2412 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:25.038944  5917 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:25.042629  5764 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.050109  2280 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.062292  2410 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.065768  4196 raft_consensus.cc:1203] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 02:15:25.067530  2475 consensus_peers.cc:485] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51dbf0fd54b14c2fa21d3e62104c9b87. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.071810  2140 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:15:25.084605  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9253de3f798f4a208d566765ed2ba54d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.087021  2280 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:25.087858  5917 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:25.103646  6142 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:25.104357  6142 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:25.104727  6142 ts_tablet_manager.cc:1367] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.135s	user 0.050s	sys 0.015s
I20240503 02:15:25.105904  6142 raft_consensus.cc:348] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.106256  6142 raft_consensus.cc:729] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:25.106586  6142 consensus_queue.cc:260] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.110817  4461 raft_consensus.cc:1203] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:25.119506  6142 ts_tablet_manager.cc:1398] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.015s	user 0.015s	sys 0.000s
I20240503 02:15:25.120644  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 7fd07044862a481aa387698017467c01-0e61eb7d9e6041b6b99d5d5328943140 received from {username='slave'} at 127.20.218.72:42451
I20240503 02:15:25.120863  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 7fd07044862a481aa387698017467c01-0e61eb7d9e6041b6b99d5d5328943140 on tablet 0e61eb7d9e6041b6b99d5d5328943140 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:25.129988  5764 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.135633  6072 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:25.136261  6072 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:25.137601  2410 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.146185  2279 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.150009  2140 catalog_manager.cc:5484] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.168675  6072 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.448s	user 0.062s	sys 0.008s
I20240503 02:15:25.169638  6072 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.169914  6072 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:25.170181  6072 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.170759  6072 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:25.172034  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:25.172341  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece on tablet 1ec063c94fe941ebb1c866b5b8935ece with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:25.182294  4593 raft_consensus.cc:1203] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:25.262751  2542 consensus_queue.cc:237] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
W20240503 02:15:25.269574  2477 consensus_peers.cc:485] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de1d29dbea945eba58f8eadb019d766. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.269960  2279 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:25.270670  2668 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:25.271262  6162 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:25.271729  6191 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:25.299875  4461 raft_consensus.cc:1203] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:25.306082  6162 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.310575  2672 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.315498  2279 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.322523  2139 catalog_manager.cc:5484] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:25.327418  4593 raft_consensus.cc:1203] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:25.394841  6072 ts_tablet_manager.cc:902] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:25.416052  6072 tablet_copy_client.cc:308] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:25.503572  2279 consensus_queue.cc:237] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:25.515762  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 9b822c691d824b2ba25e967535c9734e from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:35157)
I20240503 02:15:25.515980  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 9b822c691d824b2ba25e967535c9734e from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:35157: session id = 7fd07044862a481aa387698017467c01-9b822c691d824b2ba25e967535c9734e
W20240503 02:15:25.517652  2214 consensus_peers.cc:485] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75ba33132e784f408f3e26882aea3d15. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.517807  4461 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:25.518832  2410 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:25.519023  2432 tablet_copy_source_session.cc:205] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:25.519872  2542 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:25.520556  6186 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:25.520887  6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:25.522815  6034 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:25.523833  6034 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:25.528227  6072 tablet_copy_client.cc:783] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:25.528535  6072 tablet_copy_client.cc:647] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:25.530467  6072 tablet_copy_client.cc:523] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:25.537037  6072 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:25.564208  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:25.581528  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:25.583158  6142 ts_tablet_manager.cc:902] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:25.600919  6291 ts_tablet_manager.cc:902] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:25.604513  6186 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.608450  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:25.611799  2542 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.612295  2410 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.612741  4461 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.619575  2139 catalog_manager.cc:5484] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.632709  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:25.692986  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:25.715409  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:25.720634  6142 tablet_copy_client.cc:308] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:25.724184  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 9253de3f798f4a208d566765ed2ba54d from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:25.724408  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 9253de3f798f4a208d566765ed2ba54d from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-9253de3f798f4a208d566765ed2ba54d
I20240503 02:15:25.727483  2562 tablet_copy_source_session.cc:205] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:25.727749  2410 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
W20240503 02:15:25.731801  2345 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b822c691d824b2ba25e967535c9734e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.734485  2668 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:25.735095  2542 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:25.739428  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:25.740754  6057 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:25.741509  6057 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:25.747579  6146 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.756003  2542 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.759850  6142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253de3f798f4a208d566765ed2ba54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:25.768555  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:25.778985  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:25.788631  6142 tablet_copy_client.cc:783] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:25.788919  6142 tablet_copy_client.cc:647] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:25.798141  6325 ts_tablet_manager.cc:902] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:25.819953  6142 tablet_copy_client.cc:523] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:25.834863  6142 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:25.842087  6072 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:25.851289  2139 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.877777  6072 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:25.878291  6072 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.341s	user 0.051s	sys 0.009s
I20240503 02:15:25.879267  6072 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.879715  6072 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:25.880115  6072 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:25.877676  2668 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:25.890516  6291 tablet_copy_client.cc:308] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:25.871080  6325 tablet_copy_client.cc:308] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:25.908998  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 2de1d29dbea945eba58f8eadb019d766 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:25.909199  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 2de1d29dbea945eba58f8eadb019d766 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-2de1d29dbea945eba58f8eadb019d766
I20240503 02:15:25.912925  2562 tablet_copy_source_session.cc:205] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:25.915460  6072 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.000s
I20240503 02:15:25.915946  6325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de1d29dbea945eba58f8eadb019d766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:25.916332  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 7fd07044862a481aa387698017467c01-9b822c691d824b2ba25e967535c9734e received from {username='slave'} at 127.20.218.72:35157
I20240503 02:15:25.916520  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 7fd07044862a481aa387698017467c01-9b822c691d824b2ba25e967535c9734e on tablet 9b822c691d824b2ba25e967535c9734e with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:25.924430  6325 tablet_copy_client.cc:783] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:25.931417  6325 tablet_copy_client.cc:647] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:25.943222  6325 tablet_copy_client.cc:523] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:25.952337  6325 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:25.962342  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:43165)
I20240503 02:15:25.962569  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 51dbf0fd54b14c2fa21d3e62104c9b87 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:43165: session id = 9e699d9f984b4cf2b8b468322759b78c-51dbf0fd54b14c2fa21d3e62104c9b87
I20240503 02:15:25.966054  2562 tablet_copy_source_session.cc:205] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:25.967752  6291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dbf0fd54b14c2fa21d3e62104c9b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:25.970901  2410 consensus_queue.cc:237] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
W20240503 02:15:25.971918  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 152288us
I20240503 02:15:25.975049  6291 tablet_copy_client.cc:783] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:25.975415  6291 tablet_copy_client.cc:647] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:25.979828  6291 tablet_copy_client.cc:523] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:25.986881  6291 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:26.028384  2542 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:26.029613  6146 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:26.037487  2348 consensus_peers.cc:485] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f7c9c29384949fb95cb0c3b765f57b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:26.038439  2279 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:26.039328  6146 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:26.111109  6142 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:26.111938  6142 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:26.112811  6142 ts_tablet_manager.cc:1367] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.278s	user 0.054s	sys 0.009s
I20240503 02:15:26.114185  6142 raft_consensus.cc:348] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:26.115820  6142 raft_consensus.cc:729] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:26.116328  6142 consensus_queue.cc:260] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:26.118052  6142 ts_tablet_manager.cc:1398] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240503 02:15:26.119870  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-9253de3f798f4a208d566765ed2ba54d received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:26.120046  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-9253de3f798f4a208d566765ed2ba54d on tablet 9253de3f798f4a208d566765ed2ba54d with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:26.125821  6379 ts_tablet_manager.cc:902] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:26.127435  2542 consensus_queue.cc:237] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:26.131991  4593 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:26.132783  6180 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:26.136895  2279 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:26.137653  6180 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:26.139250  6146 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:15:26.148561  2478 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9253de3f798f4a208d566765ed2ba54d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:26.149049  2410 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:26.165925  2279 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.181090  2139 catalog_manager.cc:5484] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.182915  6393 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:15:26.198191  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 46870.2us
I20240503 02:15:26.202997  6396 ts_tablet_manager.cc:902] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:26.213874  6391 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:26.222290  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 73093.1us
I20240503 02:15:26.224243  4593 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.231715  6379 tablet_copy_client.cc:308] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:26.238977  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 75ba33132e784f408f3e26882aea3d15 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:34139)
I20240503 02:15:26.239238  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 75ba33132e784f408f3e26882aea3d15 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:34139: session id = 9e699d9f984b4cf2b8b468322759b78c-75ba33132e784f408f3e26882aea3d15
I20240503 02:15:26.239754  2541 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.242228  2301 tablet_copy_source_session.cc:205] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:26.246572  2139 catalog_manager.cc:5484] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 reported cstate change: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.246294  2410 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.267184  6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ba33132e784f408f3e26882aea3d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:26.273864  6396 tablet_copy_client.cc:308] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:26.274684  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 9b822c691d824b2ba25e967535c9734e from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:60421)
I20240503 02:15:26.274866  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 9b822c691d824b2ba25e967535c9734e from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:60421: session id = 9e699d9f984b4cf2b8b468322759b78c-9b822c691d824b2ba25e967535c9734e
I20240503 02:15:26.278762  2432 tablet_copy_source_session.cc:205] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:26.281271  6396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:26.299502  2279 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:26.303270  6396 tablet_copy_client.cc:783] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:26.303575  6396 tablet_copy_client.cc:647] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:26.304376  6379 tablet_copy_client.cc:783] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:26.304641  6379 tablet_copy_client.cc:647] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:26.306947  6379 tablet_copy_client.cc:523] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:26.311843  4593 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:26.312584  6396 tablet_copy_client.cc:523] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:26.324133  6396 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:26.346514  6379 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:26.352319  6291 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:26.397908  6291 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:26.398300  6291 ts_tablet_manager.cc:1367] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.412s	user 0.113s	sys 0.095s
I20240503 02:15:26.399325  6291 raft_consensus.cc:348] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.399575  6291 raft_consensus.cc:729] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:26.399889  6291 consensus_queue.cc:260] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.420804  6291 ts_tablet_manager.cc:1398] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.005s
I20240503 02:15:26.426888  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-51dbf0fd54b14c2fa21d3e62104c9b87 received from {username='slave'} at 127.20.218.70:43165
I20240503 02:15:26.427062  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-51dbf0fd54b14c2fa21d3e62104c9b87 on tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:26.518837  6456 ts_tablet_manager.cc:902] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:26.522717  6325 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:26.530165  6325 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:26.531129  6325 ts_tablet_manager.cc:1367] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.579s	user 0.063s	sys 0.005s
I20240503 02:15:26.533797  2410 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:26.536357  6325 raft_consensus.cc:348] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:26.537551  2668 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:26.537648  6325 raft_consensus.cc:729] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:26.538427  6325 consensus_queue.cc:260] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:26.543743  6325 ts_tablet_manager.cc:1398] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240503 02:15:26.547219  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-2de1d29dbea945eba58f8eadb019d766 received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:26.547377  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-2de1d29dbea945eba58f8eadb019d766 on tablet 2de1d29dbea945eba58f8eadb019d766 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:26.553046  2279 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:26.555783  6057 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:26.601604  6057 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:26.602435  6484 ts_tablet_manager.cc:902] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:26.608188  4327 raft_consensus.cc:1203] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:26.614004  6456 tablet_copy_client.cc:308] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:26.636552  6396 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:26.653697  6396 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:26.654220  6396 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.330s	user 0.074s	sys 0.035s
I20240503 02:15:26.655611  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 9f7c9c29384949fb95cb0c3b765f57b0 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56697)
I20240503 02:15:26.655795  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 9f7c9c29384949fb95cb0c3b765f57b0 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56697: session id = 408af0ab5520465e942cc714bfc6c73b-9f7c9c29384949fb95cb0c3b765f57b0
I20240503 02:15:26.664624  2432 tablet_copy_source_session.cc:205] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:26.667163  6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7c9c29384949fb95cb0c3b765f57b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:26.655189  6396 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.669878  6396 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:26.670323  6396 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.675009  6396 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.021s	user 0.001s	sys 0.001s
I20240503 02:15:26.684064  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-9b822c691d824b2ba25e967535c9734e received from {username='slave'} at 127.20.218.70:60421
I20240503 02:15:26.684329  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-9b822c691d824b2ba25e967535c9734e on tablet 9b822c691d824b2ba25e967535c9734e with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:26.690374  6057 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:26.700703  6456 tablet_copy_client.cc:783] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:26.701047  6456 tablet_copy_client.cc:647] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:26.710304  6379 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:26.710808  6379 tablet_bootstrap.cc:492] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:26.711136  6379 ts_tablet_manager.cc:1367] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.365s	user 0.070s	sys 0.001s
I20240503 02:15:26.712945  2279 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:26.713961  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:26.713855  6379 raft_consensus.cc:348] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.714135  6379 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:26.714141  2136 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:26.720225  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:26.721787  2668 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:26.725564  6379 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:26.726418  6379 ts_tablet_manager.cc:1398] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240503 02:15:26.726611  6456 tablet_copy_client.cc:523] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:26.730702  6456 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
W20240503 02:15:26.754875  2347 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 771a4190142b4fd5a9479417122c0881. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:26.761876  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:26.764132  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:26.776603  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:26.804479  6484 tablet_copy_client.cc:308] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:26.808369  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:26.812817  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:26.830838  4593 raft_consensus.cc:1203] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:26.832068  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-75ba33132e784f408f3e26882aea3d15 received from {username='slave'} at 127.20.218.70:34139
I20240503 02:15:26.832273  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-75ba33132e784f408f3e26882aea3d15 on tablet 75ba33132e784f408f3e26882aea3d15 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:26.863341  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:26.926527  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 9253de3f798f4a208d566765ed2ba54d from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:35399)
I20240503 02:15:26.926802  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 9253de3f798f4a208d566765ed2ba54d from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:35399: session id = 408af0ab5520465e942cc714bfc6c73b-9253de3f798f4a208d566765ed2ba54d
I20240503 02:15:26.936285  2562 tablet_copy_source_session.cc:205] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:26.938983  6484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9253de3f798f4a208d566765ed2ba54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:26.947295  6484 tablet_copy_client.cc:783] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:26.947616  6484 tablet_copy_client.cc:647] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:26.959412  6533 ts_tablet_manager.cc:902] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:26.970474  4327 raft_consensus.cc:1203] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:26.979480  6484 tablet_copy_client.cc:523] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:26.983081  6533 tablet_copy_client.cc:308] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:27.014407  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 771a4190142b4fd5a9479417122c0881 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:49243)
I20240503 02:15:27.014690  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 771a4190142b4fd5a9479417122c0881 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:49243: session id = 1d36807868fe4a82bca42c25831ce8a2-771a4190142b4fd5a9479417122c0881
I20240503 02:15:27.019162  2432 tablet_copy_source_session.cc:205] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:27.024540  6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:27.051811  6533 tablet_copy_client.cc:783] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:27.053211  6533 tablet_copy_client.cc:647] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:27.057001  6533 tablet_copy_client.cc:523] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:27.070331  6484 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:27.073647  6533 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:27.108633  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:27.126642  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:27.127182  4327 raft_consensus.cc:1203] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:27.140167  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:27.216950  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:27.250612  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:27.265774  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:27.282387  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:27.305009  6533 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:27.307715  6533 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:27.309623  6533 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.235s	user 0.057s	sys 0.020s
I20240503 02:15:27.310359  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:27.311525  6533 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:27.319499  6533 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:27.320442  6533 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:27.329265  6533 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20240503 02:15:27.331004  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-771a4190142b4fd5a9479417122c0881 received from {username='slave'} at 127.20.218.69:49243
I20240503 02:15:27.331182  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-771a4190142b4fd5a9479417122c0881 on tablet 771a4190142b4fd5a9479417122c0881 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:27.346861  2541 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
W20240503 02:15:27.375313  2477 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 592cf4d5b35e49bfa4173343d64367f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:27.376035  2668 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:27.380594  6392 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:27.389556  2410 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:27.390352  6393 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:27.422482  6393 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:27.426741  2668 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:27.437685  2140 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:27.440157  2410 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:27.440723  4196 raft_consensus.cc:1203] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:27.602597  6456 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:27.603132  6456 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:27.604386  6456 ts_tablet_manager.cc:1367] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.874s	user 0.053s	sys 0.008s
I20240503 02:15:27.609115  6456 raft_consensus.cc:348] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:27.609544  6456 raft_consensus.cc:729] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:27.609865  6456 consensus_queue.cc:260] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:27.619661  6456 ts_tablet_manager.cc:1398] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.006s
I20240503 02:15:27.620754  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-9f7c9c29384949fb95cb0c3b765f57b0 received from {username='slave'} at 127.20.218.71:56697
I20240503 02:15:27.620916  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-9f7c9c29384949fb95cb0c3b765f57b0 on tablet 9f7c9c29384949fb95cb0c3b765f57b0 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:27.634884  6484 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:27.635490  6484 tablet_bootstrap.cc:492] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:27.635911  6484 ts_tablet_manager.cc:1367] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.569s	user 0.053s	sys 0.007s
I20240503 02:15:27.637090  6484 raft_consensus.cc:348] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:27.637461  6484 raft_consensus.cc:729] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:27.637893  6484 consensus_queue.cc:260] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:27.638749  6484 ts_tablet_manager.cc:1398] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:27.639768  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-9253de3f798f4a208d566765ed2ba54d received from {username='slave'} at 127.20.218.71:35399
I20240503 02:15:27.639932  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-9253de3f798f4a208d566765ed2ba54d on tablet 9253de3f798f4a208d566765ed2ba54d with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:27.866606  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:15:27.912271  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 54162.8us
I20240503 02:15:27.918785  2541 consensus_queue.cc:237] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:27.948088  2280 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:27.948661  2672 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:27.949587  6392 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:27.952157  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:27.952772  4592 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:27.969691  6392 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:27.970160  6393 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:27.975813  6655 ts_tablet_manager.cc:902] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:27.988544  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:27.991232  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:28.010629  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.012917  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:28.021785  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:28.022083  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.042058  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.058504  6392 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:15:28.058509  2475 consensus_peers.cc:485] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2de1d29dbea945eba58f8eadb019d766. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:28.064693  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.077373  2280 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:28.077631  2668 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:28.088644  2140 catalog_manager.cc:5484] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:15:28.122262  6665 log.cc:924] Time spent T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20240503 02:15:28.123414  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.126519  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:28.137646  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:28.138796  4592 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:28.141650  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:28.152731  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.169674  4461 raft_consensus.cc:1203] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:28.181380  4460 raft_consensus.cc:1203] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:28.215289  6655 tablet_copy_client.cc:308] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:28.216701  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:28.216904  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-592cf4d5b35e49bfa4173343d64367f4
I20240503 02:15:28.219899  2562 tablet_copy_source_session.cc:205] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:28.226284  6655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:28.294449  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:28.319712  6655 tablet_copy_client.cc:783] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:28.324069  6655 tablet_copy_client.cc:647] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:28.338685  6655 tablet_copy_client.cc:523] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:28.358386  2541 consensus_queue.cc:237] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:28.364714  6655 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:28.391384  6708 ts_tablet_manager.cc:902] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:28.408869  2280 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:28.409819  6393 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:28.432924  2672 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:28.435406  6672 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:15:28.458149  2477 consensus_peers.cc:485] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae239d73f0bc4d9aac025e40489e81a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:28.499871  6392 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.508716  2280 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.514088  2672 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.516041  2139 catalog_manager.cc:5484] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.530777  6708 tablet_copy_client.cc:308] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:28.531525  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 2de1d29dbea945eba58f8eadb019d766 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:43165)
I20240503 02:15:28.531747  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 2de1d29dbea945eba58f8eadb019d766 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:43165: session id = 9e699d9f984b4cf2b8b468322759b78c-2de1d29dbea945eba58f8eadb019d766
I20240503 02:15:28.535868  2562 tablet_copy_source_session.cc:205] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:28.540022  6708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de1d29dbea945eba58f8eadb019d766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:28.557457  6708 tablet_copy_client.cc:783] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:28.557818  6708 tablet_copy_client.cc:647] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:28.565268  6708 tablet_copy_client.cc:523] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:28.570132  2410 consensus_queue.cc:237] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:28.580760  6708 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:28.583657  4327 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:28.585101  6348 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:28.586766  2672 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:28.587441  2541 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:28.588677  6057 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:28.589366  6057 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:28.590934  2348 consensus_peers.cc:485] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f084f1783d4d477e9c3791984e681f4c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:28.603838  2668 consensus_queue.cc:237] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:28.633347  6057 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:28.636327  2540 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:28.645108  2139 catalog_manager.cc:5484] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:15:28.646961  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 62561.4us
W20240503 02:15:28.648674  2608 consensus_peers.cc:485] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e248f0cb764d229802851c4e73cc96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:28.649854  2540 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:28.651247  2412 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:28.653992  5671 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:28.657581  5671 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:28.661105  2668 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:28.672857  6655 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:28.673446  6655 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:28.673763  6655 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.309s	user 0.061s	sys 0.008s
I20240503 02:15:28.676311  6655 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:28.678401  6655 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:28.678825  6655 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:28.686910  6655 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.008s
I20240503 02:15:28.687876  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-592cf4d5b35e49bfa4173343d64367f4 received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:28.688078  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-592cf4d5b35e49bfa4173343d64367f4 on tablet 592cf4d5b35e49bfa4173343d64367f4 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:28.691524  4327 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:28.694041  5671 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.704310  2412 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.723116  2140 catalog_manager.cc:5484] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.729122  2540 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:28.878515  6708 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:28.879156  6708 tablet_bootstrap.cc:492] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:28.879550  6708 ts_tablet_manager.cc:1367] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.299s	user 0.055s	sys 0.007s
I20240503 02:15:28.880703  6708 raft_consensus.cc:348] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:28.881099  6708 raft_consensus.cc:729] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:28.881907  6708 consensus_queue.cc:260] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:28.883731  6708 ts_tablet_manager.cc:1398] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:15:28.891991  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-2de1d29dbea945eba58f8eadb019d766 received from {username='slave'} at 127.20.218.70:43165
I20240503 02:15:28.892206  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-2de1d29dbea945eba58f8eadb019d766 on tablet 2de1d29dbea945eba58f8eadb019d766 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:28.923547  2412 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:28.939343  4327 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:28.940412  6348 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:28.952322  4592 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:28.954407  6348 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:29.006502  4591 raft_consensus.cc:1203] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240503 02:15:29.057246  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b822c691d824b2ba25e967535c9734e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:29.061899  2668 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:29.062852  6348 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:29.068557  2541 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:29.069599  6057 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:29.129034  6817 ts_tablet_manager.cc:902] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:29.137591  6818 ts_tablet_manager.cc:902] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:29.139531  6057 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.150039  2668 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.162040  4592 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.165350  4327 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.165810  2541 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.178547  6832 ts_tablet_manager.cc:902] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:29.180013  2140 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 5 to 6, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.188107  6832 tablet_copy_client.cc:308] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:29.188932  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 43e248f0cb764d229802851c4e73cc96 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:44663)
I20240503 02:15:29.189182  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 43e248f0cb764d229802851c4e73cc96 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:44663: session id = 1d36807868fe4a82bca42c25831ce8a2-43e248f0cb764d229802851c4e73cc96
I20240503 02:15:29.192469  2693 tablet_copy_source_session.cc:205] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:29.195422  6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e248f0cb764d229802851c4e73cc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:29.243808  6832 tablet_copy_client.cc:783] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:29.244911  6832 tablet_copy_client.cc:647] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:29.251917  6832 tablet_copy_client.cc:523] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:29.256253  6818 tablet_copy_client.cc:308] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:29.257246  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet f084f1783d4d477e9c3791984e681f4c from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56697)
I20240503 02:15:29.258118  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet f084f1783d4d477e9c3791984e681f4c from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56697: session id = 408af0ab5520465e942cc714bfc6c73b-f084f1783d4d477e9c3791984e681f4c
I20240503 02:15:29.260768  2540 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:29.261785  2432 tablet_copy_source_session.cc:205] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:29.262830  6832 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:29.268990  6817 tablet_copy_client.cc:308] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
W20240503 02:15:29.281164  2475 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 592cf4d5b35e49bfa4173343d64367f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:29.281670  4592 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.282732  2410 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.283000  2541 consensus_queue.cc:237] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:29.283577  6392 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.286688  2668 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.288009  6393 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.288564  6393 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.294873  6818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f084f1783d4d477e9c3791984e681f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:29.301884  2279 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:29.302670  6392 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:29.305027  2672 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:29.306336  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 913d0aa014ad4bcaac905b5fdaa296d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:29.306546  6392 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:29.324466  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet ae239d73f0bc4d9aac025e40489e81a6 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:57071)
I20240503 02:15:29.324681  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet ae239d73f0bc4d9aac025e40489e81a6 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:57071: session id = 1d36807868fe4a82bca42c25831ce8a2-ae239d73f0bc4d9aac025e40489e81a6
I20240503 02:15:29.327693  2562 tablet_copy_source_session.cc:205] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:29.329771  6817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae239d73f0bc4d9aac025e40489e81a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:29.343942  4327 raft_consensus.cc:1203] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:29.349169  6817 tablet_copy_client.cc:783] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:29.349809  6817 tablet_copy_client.cc:647] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:29.350281  6818 tablet_copy_client.cc:783] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:29.353715  6818 tablet_copy_client.cc:647] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:29.356844  6818 tablet_copy_client.cc:523] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:29.357579  6817 tablet_copy_client.cc:523] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:29.361181  6392 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.364737  6818 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:29.366006  2669 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.371675  6817 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:29.375707  2139 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.379911  6393 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.386243  2668 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.388794  2412 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.393836  2136 catalog_manager.cc:5484] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.403133  4591 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.407824  2279 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:29.506700  6832 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:29.511039  6832 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:29.511428  6832 ts_tablet_manager.cc:1367] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.249s	user 0.065s	sys 0.019s
I20240503 02:15:29.513106  6832 raft_consensus.cc:348] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:29.513433  6832 raft_consensus.cc:729] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:29.515131  6832 consensus_queue.cc:260] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:29.522267  6832 ts_tablet_manager.cc:1398] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20240503 02:15:29.531010  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-43e248f0cb764d229802851c4e73cc96 received from {username='slave'} at 127.20.218.69:44663
I20240503 02:15:29.531221  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-43e248f0cb764d229802851c4e73cc96 on tablet 43e248f0cb764d229802851c4e73cc96 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:29.580261  2668 consensus_queue.cc:237] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:29.583751  4196 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.584535  6905 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.586016  2540 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.587134  6905 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.596246  2410 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.596983  6905 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:29.597729  2606 consensus_peers.cc:485] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e248f0cb764d229802851c4e73cc96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:29.600100  6766 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.600616  2668 consensus_queue.cc:237] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:29.607940  2540 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.607846  2410 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.612720  2139 catalog_manager.cc:5484] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.618175  6818 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:29.619187  6818 tablet_bootstrap.cc:492] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:29.619840  6818 ts_tablet_manager.cc:1367] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.255s	user 0.075s	sys 0.028s
I20240503 02:15:29.621069  6818 raft_consensus.cc:348] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:29.621625  6818 raft_consensus.cc:729] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:29.622062  6818 consensus_queue.cc:260] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
W20240503 02:15:29.635520  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1949a5c7cc741b19da021eccb7d0dd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:29.638032  4195 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.639082  6905 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.647135  6832 ts_tablet_manager.cc:902] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:29.653218  2410 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.654215  6905 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.651964  4196 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:29.698498  6817 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:29.699172  6817 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:29.699563  6817 ts_tablet_manager.cc:1367] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.328s	user 0.071s	sys 0.004s
I20240503 02:15:29.701043  6817 raft_consensus.cc:348] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:29.709540  6817 raft_consensus.cc:729] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:29.710628  6817 consensus_queue.cc:260] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:29.711437  6817 ts_tablet_manager.cc:1398] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.000s
I20240503 02:15:29.713447  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-ae239d73f0bc4d9aac025e40489e81a6 received from {username='slave'} at 127.20.218.69:57071
I20240503 02:15:29.713596  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-ae239d73f0bc4d9aac025e40489e81a6 on tablet ae239d73f0bc4d9aac025e40489e81a6 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:29.728034  2540 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:29.729635  6905 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:29.786403  6832 tablet_copy_client.cc:308] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:29.787483  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 9b822c691d824b2ba25e967535c9734e from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:49243)
I20240503 02:15:29.787714  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 9b822c691d824b2ba25e967535c9734e from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:49243: session id = 1d36807868fe4a82bca42c25831ce8a2-9b822c691d824b2ba25e967535c9734e
I20240503 02:15:29.789383  6818 ts_tablet_manager.cc:1398] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.169s	user 0.026s	sys 0.039s
I20240503 02:15:29.790524  2431 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-f084f1783d4d477e9c3791984e681f4c received from {username='slave'} at 127.20.218.71:56697
I20240503 02:15:29.790932  2431 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-f084f1783d4d477e9c3791984e681f4c on tablet f084f1783d4d477e9c3791984e681f4c with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:29.790686  2432 tablet_copy_source_session.cc:205] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:29.794838  6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b822c691d824b2ba25e967535c9734e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:29.809749  6934 ts_tablet_manager.cc:902] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:29.813190  6766 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:29.813894  6832 tablet_copy_client.cc:783] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:29.814155  6832 tablet_copy_client.cc:647] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:29.815824  6832 tablet_copy_client.cc:523] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:29.819968  6832 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:29.828286  2542 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:29.836230  2410 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:29.836710  2139 catalog_manager.cc:5484] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
W20240503 02:15:29.851840  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 160596us
I20240503 02:15:29.856654  6817 ts_tablet_manager.cc:902] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:29.901593  4196 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:29.948567  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:29.999598  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:30.001116  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:30.017616  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:30.113811  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:30.142158  6999 ts_tablet_manager.cc:902] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:30.143431  7000 ts_tablet_manager.cc:902] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:15:30.167699  4195 raft_consensus.cc:1203] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:30.177178  6934 tablet_copy_client.cc:308] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:30.178418  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:43165)
I20240503 02:15:30.178664  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:43165: session id = 9e699d9f984b4cf2b8b468322759b78c-592cf4d5b35e49bfa4173343d64367f4
I20240503 02:15:30.182859  6999 tablet_copy_client.cc:308] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:30.184128  2562 tablet_copy_source_session.cc:205] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:30.184285  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet a1949a5c7cc741b19da021eccb7d0dd1 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:42451)
I20240503 02:15:30.184540  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet a1949a5c7cc741b19da021eccb7d0dd1 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:42451: session id = 7fd07044862a481aa387698017467c01-a1949a5c7cc741b19da021eccb7d0dd1
I20240503 02:15:30.189255  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:30.190824  2693 tablet_copy_source_session.cc:205] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:30.217232  6934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:30.220439  6999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1949a5c7cc741b19da021eccb7d0dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:30.242599  7000 tablet_copy_client.cc:308] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:15:30.243598  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 43e248f0cb764d229802851c4e73cc96 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:52945)
I20240503 02:15:30.243897  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 43e248f0cb764d229802851c4e73cc96 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:52945: session id = 9e699d9f984b4cf2b8b468322759b78c-43e248f0cb764d229802851c4e73cc96
I20240503 02:15:30.247175  2693 tablet_copy_source_session.cc:205] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:30.248875  7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e248f0cb764d229802851c4e73cc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:30.251132  4460 raft_consensus.cc:1203] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:30.266093  6832 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:30.266757  6832 tablet_bootstrap.cc:492] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:30.267051  6832 ts_tablet_manager.cc:1367] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.447s	user 0.099s	sys 0.009s
I20240503 02:15:30.273181  6832 raft_consensus.cc:348] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:30.273538  6832 raft_consensus.cc:729] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:30.273957  6832 consensus_queue.cc:260] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:30.274410  6999 tablet_copy_client.cc:783] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:30.274791  6999 tablet_copy_client.cc:647] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:30.274992  7000 tablet_copy_client.cc:783] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:30.279754  6999 tablet_copy_client.cc:523] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:30.280735  7000 tablet_copy_client.cc:647] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:30.294210  6832 ts_tablet_manager.cc:1398] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.000s
I20240503 02:15:30.295250  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-9b822c691d824b2ba25e967535c9734e received from {username='slave'} at 127.20.218.69:49243
I20240503 02:15:30.295396  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-9b822c691d824b2ba25e967535c9734e on tablet 9b822c691d824b2ba25e967535c9734e with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:30.300283  7000 tablet_copy_client.cc:523] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:30.304730  7000 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:30.310878  6817 tablet_copy_client.cc:308] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:30.312727  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 913d0aa014ad4bcaac905b5fdaa296d2 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:57071)
I20240503 02:15:30.313043  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 913d0aa014ad4bcaac905b5fdaa296d2 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:57071: session id = 1d36807868fe4a82bca42c25831ce8a2-913d0aa014ad4bcaac905b5fdaa296d2
I20240503 02:15:30.317006  6999 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:30.320480  2562 tablet_copy_source_session.cc:205] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:30.322121  6817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d0aa014ad4bcaac905b5fdaa296d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:30.329736  6817 tablet_copy_client.cc:783] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:30.330008  6817 tablet_copy_client.cc:647] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
W20240503 02:15:30.337561  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 56344.3us
I20240503 02:15:30.337642  6817 tablet_copy_client.cc:523] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:30.342940  6817 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:30.364701  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:30.413357  4195 raft_consensus.cc:1203] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240503 02:15:30.488842  6817 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:30.489459  6817 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:30.489815  6817 ts_tablet_manager.cc:1367] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.147s	user 0.059s	sys 0.001s
I20240503 02:15:30.493119  6817 raft_consensus.cc:348] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:30.493419  6817 raft_consensus.cc:729] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:30.493745  6817 consensus_queue.cc:260] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:30.494556  6817 ts_tablet_manager.cc:1398] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:15:30.513509  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-913d0aa014ad4bcaac905b5fdaa296d2 received from {username='slave'} at 127.20.218.69:57071
I20240503 02:15:30.513847  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-913d0aa014ad4bcaac905b5fdaa296d2 on tablet 913d0aa014ad4bcaac905b5fdaa296d2 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:30.535763  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:30.538298  6999 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:30.538969  6999 tablet_bootstrap.cc:492] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:30.539353  6999 ts_tablet_manager.cc:1367] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.223s	user 0.058s	sys 0.017s
I20240503 02:15:30.540485  6999 raft_consensus.cc:348] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:30.540868  6999 raft_consensus.cc:729] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:30.541213  6999 consensus_queue.cc:260] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:30.572156  6999 ts_tablet_manager.cc:1398] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.033s	user 0.015s	sys 0.007s
I20240503 02:15:30.579890  6934 tablet_copy_client.cc:783] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:30.580276  6934 tablet_copy_client.cc:647] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:30.583360  6934 tablet_copy_client.cc:523] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:30.589426  2279 consensus_queue.cc:237] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:30.603634  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 7fd07044862a481aa387698017467c01-a1949a5c7cc741b19da021eccb7d0dd1 received from {username='slave'} at 127.20.218.72:42451
I20240503 02:15:30.603864  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 7fd07044862a481aa387698017467c01-a1949a5c7cc741b19da021eccb7d0dd1 on tablet a1949a5c7cc741b19da021eccb7d0dd1 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:30.606979  2540 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:30.607792  6034 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:30.614009  7000 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:30.614557  7000 tablet_bootstrap.cc:492] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:30.614859  7000 ts_tablet_manager.cc:1367] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.310s	user 0.098s	sys 0.054s
I20240503 02:15:30.615698  7000 raft_consensus.cc:348] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:30.616006  7000 raft_consensus.cc:729] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
W20240503 02:15:30.616026  2216 consensus_peers.cc:485] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a62b3ef44e94e92b0feed9c228f5702. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:30.616233  7000 consensus_queue.cc:260] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:30.617758  2668 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:30.618937  6034 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:30.624315  7000 ts_tablet_manager.cc:1398] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.007s
I20240503 02:15:30.625662  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-43e248f0cb764d229802851c4e73cc96 received from {username='slave'} at 127.20.218.70:52945
I20240503 02:15:30.625854  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-43e248f0cb764d229802851c4e73cc96 on tablet 43e248f0cb764d229802851c4e73cc96 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:30.637174  6034 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:30.641098  6934 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:30.645695  2136 catalog_manager.cc:5484] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:30.643656  2668 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:30.652542  2540 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:30.745877  2412 consensus_queue.cc:237] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:30.778663  2542 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:15:30.780906  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7ab8b0c8db54565a6066b9a7d1569d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:30.783838  2279 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:15:30.786316  6057 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:30.791417  6057 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:15:30.848604  6057 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:30.852911  2279 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:30.864276  2139 catalog_manager.cc:5484] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:30.891350  2540 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:30.919123  4591 raft_consensus.cc:1203] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:31.062108  6934 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:31.062992  6934 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:31.063458  6934 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.422s	user 0.048s	sys 0.013s
I20240503 02:15:31.064718  6934 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:31.065217  6934 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:31.065711  6934 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:31.066615  6934 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:15:31.067950  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-592cf4d5b35e49bfa4173343d64367f4 received from {username='slave'} at 127.20.218.70:43165
I20240503 02:15:31.068125  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-592cf4d5b35e49bfa4173343d64367f4 on tablet 592cf4d5b35e49bfa4173343d64367f4 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:31.083007  4195 raft_consensus.cc:1203] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:31.110414  4326 raft_consensus.cc:1203] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:31.155704  2540 consensus_queue.cc:237] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:31.159333  2668 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:31.159628  4195 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:31.160293  6881 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:15:31.164093  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 913d0aa014ad4bcaac905b5fdaa296d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:31.164366  6881 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:31.184099  4195 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:31.182569  6881 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:31.191658  2140 catalog_manager.cc:5484] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:31.219496  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:31.223302  2668 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:31.224229  2280 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:31.226500  6392 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:31.242322  4326 raft_consensus.cc:1203] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:31.277825  7149 ts_tablet_manager.cc:902] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:31.288933  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:31.308398  2280 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:31.321512  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.351549  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:31.356882  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.369829  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.389046  7149 tablet_copy_client.cc:308] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:31.390272  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 7a62b3ef44e94e92b0feed9c228f5702 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:35505)
I20240503 02:15:31.390516  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 7a62b3ef44e94e92b0feed9c228f5702 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:35505: session id = 1d36807868fe4a82bca42c25831ce8a2-7a62b3ef44e94e92b0feed9c228f5702
I20240503 02:15:31.394243  2301 tablet_copy_source_session.cc:205] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:31.395848  7149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a62b3ef44e94e92b0feed9c228f5702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:31.419297  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.442181  7173 ts_tablet_manager.cc:902] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:31.443428  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:31.492098  7173 tablet_copy_client.cc:308] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:31.493225  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet a7ab8b0c8db54565a6066b9a7d1569d8 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:35157)
I20240503 02:15:31.493441  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet a7ab8b0c8db54565a6066b9a7d1569d8 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:35157: session id = 7fd07044862a481aa387698017467c01-a7ab8b0c8db54565a6066b9a7d1569d8
I20240503 02:15:31.496604  2432 tablet_copy_source_session.cc:205] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:31.508517  7149 tablet_copy_client.cc:783] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:31.508857  7149 tablet_copy_client.cc:647] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:31.511389  7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ab8b0c8db54565a6066b9a7d1569d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:31.523205  7149 tablet_copy_client.cc:523] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:31.552634  7173 tablet_copy_client.cc:783] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:31.554813  7173 tablet_copy_client.cc:647] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:31.560632  7173 tablet_copy_client.cc:523] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:31.562978  2280 consensus_queue.cc:237] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:31.567423  2668 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:15:31.568713  2217 consensus_peers.cc:485] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a62b3ef44e94e92b0feed9c228f5702. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:15:31.568722  2216 consensus_peers.cc:485] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:31.571173  7212 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:31.573717  2540 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:31.574151  7173 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:31.574851  6034 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:31.579563  7149 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:31.581849  6034 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:31.589223  2668 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:31.595871  2136 catalog_manager.cc:5484] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:31.602608  2540 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:31.630744  7213 ts_tablet_manager.cc:902] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:31.727528  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:31.728422  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:31.749027  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.774036  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:31.795075  7213 tablet_copy_client.cc:308] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:31.796048  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 913d0aa014ad4bcaac905b5fdaa296d2 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:31.796258  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 913d0aa014ad4bcaac905b5fdaa296d2 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-913d0aa014ad4bcaac905b5fdaa296d2
I20240503 02:15:31.799422  2562 tablet_copy_source_session.cc:205] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:31.801708  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.802815  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.802956  7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913d0aa014ad4bcaac905b5fdaa296d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:31.832203  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:31.850180  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:31.946522  7149 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:31.947067  7149 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:31.947281  7213 tablet_copy_client.cc:783] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:31.947413  7149 ts_tablet_manager.cc:1367] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.368s	user 0.055s	sys 0.004s
I20240503 02:15:31.947567  7213 tablet_copy_client.cc:647] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:31.948374  7149 raft_consensus.cc:348] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:31.948647  7149 raft_consensus.cc:729] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:31.949880  7213 tablet_copy_client.cc:523] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:31.955713  7149 consensus_queue.cc:260] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:31.963167  7149 ts_tablet_manager.cc:1398] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.002s
I20240503 02:15:31.973436  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-7a62b3ef44e94e92b0feed9c228f5702 received from {username='slave'} at 127.20.218.69:35505
I20240503 02:15:31.973640  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-7a62b3ef44e94e92b0feed9c228f5702 on tablet 7a62b3ef44e94e92b0feed9c228f5702 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:31.977006  7213 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:32.031767  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:32.049230  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:32.062247  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:32.073889  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:32.084443  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:32.088331  7283 ts_tablet_manager.cc:902] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:32.093623  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:32.145493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:32.165176  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:32.171038  7283 tablet_copy_client.cc:308] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:32.172178  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 7a62b3ef44e94e92b0feed9c228f5702 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56489)
I20240503 02:15:32.172418  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 7a62b3ef44e94e92b0feed9c228f5702 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56489: session id = 408af0ab5520465e942cc714bfc6c73b-7a62b3ef44e94e92b0feed9c228f5702
I20240503 02:15:32.176059  2301 tablet_copy_source_session.cc:205] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:32.178539  7283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a62b3ef44e94e92b0feed9c228f5702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:32.195659  7283 tablet_copy_client.cc:783] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:32.195353  4195 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:32.196262  7283 tablet_copy_client.cc:647] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:32.200461  7283 tablet_copy_client.cc:523] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:32.204165  7173 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:32.236830  7173 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:32.237262  7173 ts_tablet_manager.cc:1367] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.663s	user 0.064s	sys 0.026s
I20240503 02:15:32.243028  7283 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:15:32.259848  7173 raft_consensus.cc:348] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:32.260241  7173 raft_consensus.cc:729] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:32.260587  7173 consensus_queue.cc:260] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:32.305137  7173 ts_tablet_manager.cc:1398] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.053s	user 0.001s	sys 0.001s
I20240503 02:15:32.306294  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 7fd07044862a481aa387698017467c01-a7ab8b0c8db54565a6066b9a7d1569d8 received from {username='slave'} at 127.20.218.72:35157
I20240503 02:15:32.306499  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 7fd07044862a481aa387698017467c01-a7ab8b0c8db54565a6066b9a7d1569d8 on tablet a7ab8b0c8db54565a6066b9a7d1569d8 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:32.413547  7213 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:32.414548  7213 tablet_bootstrap.cc:492] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:32.414888  7213 ts_tablet_manager.cc:1367] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.438s	user 0.056s	sys 0.005s
I20240503 02:15:32.415748  7213 raft_consensus.cc:348] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:32.416033  7213 raft_consensus.cc:729] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:32.416288  7213 consensus_queue.cc:260] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:32.422065  7213 ts_tablet_manager.cc:1398] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240503 02:15:32.423067  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-913d0aa014ad4bcaac905b5fdaa296d2 received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:32.423244  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-913d0aa014ad4bcaac905b5fdaa296d2 on tablet 913d0aa014ad4bcaac905b5fdaa296d2 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:32.476198  7283 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:32.476935  7283 tablet_bootstrap.cc:492] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:15:32.477447  7283 ts_tablet_manager.cc:1367] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.235s	user 0.064s	sys 0.011s
I20240503 02:15:32.480298  7283 raft_consensus.cc:348] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:32.480748  7283 raft_consensus.cc:729] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:15:32.481118  7283 consensus_queue.cc:260] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:32.488981  7283 ts_tablet_manager.cc:1398] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20240503 02:15:32.491897  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-7a62b3ef44e94e92b0feed9c228f5702 received from {username='slave'} at 127.20.218.71:56489
I20240503 02:15:32.492064  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-7a62b3ef44e94e92b0feed9c228f5702 on tablet 7a62b3ef44e94e92b0feed9c228f5702 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:32.528656  4460 raft_consensus.cc:1203] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:32.599803  4591 raft_consensus.cc:1203] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:15:32.682869  4591 raft_consensus.cc:1203] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:15:32.748198  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:32.833827  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:32.837688  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:32.924160  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:32.950788  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:32.998934  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.013082  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.015427  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.035413  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.045627  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.050702  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:33.081254  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:33.094060  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.136045  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.139113  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:33.186103  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:33.550434  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.553313  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:33.574787  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:33.584156  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:33.602813  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.718647  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:33.719748  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.720777  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:33.748054  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.904181  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:33.910189  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:33.929383  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:33.961549  2609 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 157148us
I20240503 02:15:33.989857  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.022188  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.050937  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:34.080969  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:34.107580  2410 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:34.117499  4195 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.117631  2280 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.118221  7440 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.119060  7440 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.154969  7440 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.158253  2279 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.164669  2136 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.197598  4196 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.248541  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a4190142b4fd5a9479417122c0881 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:34.288316  7581 tablet_replica.cc:330] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:34.302845  7581 raft_consensus.cc:2229] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:34.310575  7581 raft_consensus.cc:2258] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:34.318747  7581 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:34.371136  7581 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:34.382923  7581 log.cc:1196] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/771a4190142b4fd5a9479417122c0881
I20240503 02:15:34.384857  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 771a4190142b4fd5a9479417122c0881 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
I20240503 02:15:34.427004  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.433248  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:34.463899  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.470933  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:34.519001  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:34.610862  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.614388  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:34.675387  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:34.869153  2540 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:34.886809  2410 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.887977  7146 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.890722  2542 consensus_queue.cc:237] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:34.894806  2279 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.900705  7539 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.952508  7146 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:34.960351  4196 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.961994  2668 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:34.962518  7663 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.965015  7663 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:34.965651  2279 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:34.969404  7663 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.970892  4196 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.974465  2140 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:34.979665  2668 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:34.980628  2410 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:35.014243  2135 catalog_manager.cc:5484] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:35.025506  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet ae239d73f0bc4d9aac025e40489e81a6 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:35.048980  7688 tablet_replica.cc:330] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:35.068756  7688 raft_consensus.cc:2229] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:35.072288  7688 raft_consensus.cc:2258] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:35.080819  4573 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece with delete_type TABLET_DATA_TOMBSTONED (TS 7fd07044862a481aa387698017467c01 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34002
I20240503 02:15:35.086717  7688 ts_tablet_manager.cc:1875] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:35.089481  7698 tablet_replica.cc:330] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: stopping tablet replica
I20240503 02:15:35.090873  7698 raft_consensus.cc:2229] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:35.091526  7698 raft_consensus.cc:2258] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:35.118585  7698 ts_tablet_manager.cc:1875] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:35.200117  7688 ts_tablet_manager.cc:1888] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:35.203267  7688 log.cc:1196] T ae239d73f0bc4d9aac025e40489e81a6 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae239d73f0bc4d9aac025e40489e81a6
I20240503 02:15:35.206228  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet ae239d73f0bc4d9aac025e40489e81a6 (table rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]) successfully deleted
I20240503 02:15:35.208909  7698 ts_tablet_manager.cc:1888] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:35.209223  7698 log.cc:1196] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/wals/1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:15:35.211547  2126 catalog_manager.cc:4830] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): tablet 1ec063c94fe941ebb1c866b5b8935ece (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
I20240503 02:15:35.229563  2279 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:35.242306  4196 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:35.243958  7212 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:35.250511  2412 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:35.251447  6034 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:35.285782  6034 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:35.291710  2410 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:35.296761  2136 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:35.297049  4195 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:35.309923  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 51224b6355ed41dbb0703b7feb73c223 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:35.349903  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:35.350191  7743 tablet_replica.cc:330] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:35.351673  7743 raft_consensus.cc:2229] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:35.352012  7743 raft_consensus.cc:2258] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:35.353904  7743 ts_tablet_manager.cc:1875] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:35.366925  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:35.410465  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:35.454149  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:35.464407  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:35.486595  7743 ts_tablet_manager.cc:1888] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:35.486796  7743 log.cc:1196] T 51224b6355ed41dbb0703b7feb73c223 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/51224b6355ed41dbb0703b7feb73c223
I20240503 02:15:35.526014  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:35.621176  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 51224b6355ed41dbb0703b7feb73c223 (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
I20240503 02:15:35.783445  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:35.789316  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:35.963133  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:35.974714  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:35.982775  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.035358  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.064726  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:36.103292  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:36.154785  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:36.161916  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:36.179584  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:36.189049  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.198446  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.204733  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:36.216293  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.247290  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.254009  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.262049  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.363505  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:36.374413  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:36.468752  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.470680  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.522399  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.536116  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:36.551569  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.584241  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:36.868281  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:36.900110  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.900396  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:36.910302  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.954627  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.962240  2542 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:15:36.963738  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:36.973986  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:36.996661  2412 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:36.997757  7539 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.018604  7539 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:37.027974  2132 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:37.028606  2412 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:15:37.057540  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:37.072085  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:37.074215  2542 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
W20240503 02:15:37.077828  2477 consensus_peers.cc:485] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:37.085191  7939 tablet_replica.cc:330] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:37.086602  7939 raft_consensus.cc:2229] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:37.087476  7939 raft_consensus.cc:2258] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:37.088485  7939 ts_tablet_manager.cc:1875] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:37.092276  2410 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:37.096482  7663 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:37.101689  7663 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.104444  2410 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.122234  2132 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.159241  7939 ts_tablet_manager.cc:1888] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:37.161224  7939 log.cc:1196] T 1ec063c94fe941ebb1c866b5b8935ece P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:15:37.162995  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 1ec063c94fe941ebb1c866b5b8935ece (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
W20240503 02:15:37.224308  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 84754.9us
I20240503 02:15:37.228607  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1ec063c94fe941ebb1c866b5b8935ece with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:37.242292  2541 consensus_queue.cc:237] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
W20240503 02:15:37.263020  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 48895.2us
I20240503 02:15:37.273118  4196 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.273746  4593 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.276001  7663 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.278817  2672 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.287851  7539 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.295801  7946 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.308549  2412 consensus_queue.cc:237] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:37.338918  4461 raft_consensus.cc:1261] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:37.339720  7440 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:37.343500  2279 raft_consensus.cc:1261] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:37.344594  7872 consensus_queue.cc:1035] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:37.366604  4591 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.369832  7663 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
W20240503 02:15:37.381347  2346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 108956us
I20240503 02:15:37.381824  2669 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.395835  2132 catalog_manager.cc:5484] T 913d0aa014ad4bcaac905b5fdaa296d2 P 7fd07044862a481aa387698017467c01 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.395844  4196 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:37.402143  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 913d0aa014ad4bcaac905b5fdaa296d2 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:37.402556  7939 tablet_replica.cc:330] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:37.402760  7939 raft_consensus.cc:2229] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:37.403056  7939 raft_consensus.cc:2258] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:37.403827  7939 ts_tablet_manager.cc:1875] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:37.413161  7939 ts_tablet_manager.cc:1888] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:37.413501  7939 log.cc:1196] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/913d0aa014ad4bcaac905b5fdaa296d2
I20240503 02:15:37.414556  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 913d0aa014ad4bcaac905b5fdaa296d2 (table rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]) successfully deleted
W20240503 02:15:37.419785  2215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 145449us
I20240503 02:15:37.422771  7872 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.429621  4460 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.438320  2132 catalog_manager.cc:5484] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.447295  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet a6e0d39087944e4ea13a17ca4af478ab with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:37.453639  2279 raft_consensus.cc:2932] T a6e0d39087944e4ea13a17ca4af478ab P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.463822  7990 tablet_replica.cc:330] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:37.469720  7990 raft_consensus.cc:2229] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:37.478791  7990 raft_consensus.cc:2258] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:37.483672  7990 ts_tablet_manager.cc:1875] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:37.500870  7990 ts_tablet_manager.cc:1888] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:37.501230  7990 log.cc:1196] T a6e0d39087944e4ea13a17ca4af478ab P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6e0d39087944e4ea13a17ca4af478ab
I20240503 02:15:37.502828  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet a6e0d39087944e4ea13a17ca4af478ab (table rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]) successfully deleted
I20240503 02:15:37.590379  2672 consensus_queue.cc:237] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:37.594820  4196 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.594866  4326 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.596331  7627 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.597138  7627 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.598048  2279 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:37.598666  2542 raft_consensus.cc:1261] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:37.599385  7956 consensus_queue.cc:1035] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:37.609836  4460 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:37.611143  7212 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:37.618575  2410 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:37.619629  7963 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:37.636126  8044 ts_tablet_manager.cc:902] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:37.642410  7956 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:37.646294  4327 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:37.655263  2542 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:37.661127  7963 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.662583  2132 catalog_manager.cc:5484] T 43e248f0cb764d229802851c4e73cc96 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:37.666946  2412 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.675684  2135 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.677515  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e248f0cb764d229802851c4e73cc96 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:37.683399  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e0b954008c4f30af414da6cb5d59c2 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:37.688844  8056 tablet_replica.cc:330] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:37.689591  4461 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:37.692019  8056 raft_consensus.cc:2229] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:37.696936  8056 raft_consensus.cc:2258] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:37.699162  8056 ts_tablet_manager.cc:1875] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:37.702822  4194 raft_consensus.cc:2932] T 43e248f0cb764d229802851c4e73cc96 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:37.725028  8044 tablet_copy_client.cc:308] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:37.725998  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:52963)
I20240503 02:15:37.726217  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:52963: session id = 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:15:37.730268  2562 tablet_copy_source_session.cc:205] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:37.731878  8057 tablet_replica.cc:330] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:37.732025  8044 ts_tablet_manager.cc:1875] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:15:37.732293  8057 raft_consensus.cc:2229] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:37.732781  8057 raft_consensus.cc:2258] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:37.734324  8057 ts_tablet_manager.cc:1875] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:37.731392  8056 ts_tablet_manager.cc:1888] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:37.741030  8056 log.cc:1196] T 43e248f0cb764d229802851c4e73cc96 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/43e248f0cb764d229802851c4e73cc96
I20240503 02:15:37.743469  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 43e248f0cb764d229802851c4e73cc96 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
I20240503 02:15:37.765014  8044 ts_tablet_manager.cc:1888] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:15:37.765453  8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:37.766599  8057 ts_tablet_manager.cc:1888] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:37.779523  8057 log.cc:1196] T 02e0b954008c4f30af414da6cb5d59c2 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/02e0b954008c4f30af414da6cb5d59c2
I20240503 02:15:37.780572  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 02e0b954008c4f30af414da6cb5d59c2 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:15:37.786160  8044 tablet_copy_client.cc:783] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:37.786500  8044 tablet_copy_client.cc:647] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:37.788857  8044 tablet_copy_client.cc:523] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:37.797876  8044 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:37.950186  8044 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:37.950760  8044 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:37.951128  8044 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.153s	user 0.060s	sys 0.005s
I20240503 02:15:37.952123  8044 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:37.952394  8044 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:37.952669  8044 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:37.968434  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:38.025895  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:38.102807  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.110908  8044 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.160s	user 0.046s	sys 0.083s
I20240503 02:15:38.116209  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece received from {username='slave'} at 127.20.218.72:52963
I20240503 02:15:38.116389  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 7fd07044862a481aa387698017467c01-1ec063c94fe941ebb1c866b5b8935ece on tablet 1ec063c94fe941ebb1c866b5b8935ece with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:38.185768  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.205255  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:38.207175  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:38.238425  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.275533  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.515682  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:38.526718  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.530768  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.588552  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:38.636322  4591 raft_consensus.cc:1203] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:15:38.701900  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.703533  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.795486  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:38.807232  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.819288  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.846618  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:38.851887  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 88073.8us
I20240503 02:15:38.884047  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.890730  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:38.922926  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:38.924856  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:38.928640  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:38.955526  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:15:38.963785  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 125924us
W20240503 02:15:38.966681  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 117442us
I20240503 02:15:38.987839  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:39.000449  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:39.017871  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.024525  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:39.057430  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:39.097718  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:39.101120  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.185505  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.354748  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:39.355953  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:39.375475  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:39.393494  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.405078  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.456491  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.463690  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:39.508649  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:39.612360  2279 consensus_queue.cc:237] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:39.616688  4461 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.617911  7963 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.622686  2672 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.623878  4194 raft_consensus.cc:1261] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.624867  8243 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.625559  8243 consensus_queue.cc:1035] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.679832  8243 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:39.683338  4460 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:39.683241  4194 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:39.706249  2669 raft_consensus.cc:2932] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:39.715399  2132 catalog_manager.cc:5484] T 7a62b3ef44e94e92b0feed9c228f5702 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:39.769088  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a62b3ef44e94e92b0feed9c228f5702 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:39.782208  8306 tablet_replica.cc:330] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:39.782543  8306 raft_consensus.cc:2229] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:39.782933  8306 raft_consensus.cc:2258] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:39.783829  8306 ts_tablet_manager.cc:1875] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:39.798202  8306 ts_tablet_manager.cc:1888] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:39.799808  8306 log.cc:1196] T 7a62b3ef44e94e92b0feed9c228f5702 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a62b3ef44e94e92b0feed9c228f5702
I20240503 02:15:39.801021  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 7a62b3ef44e94e92b0feed9c228f5702 (table rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]) successfully deleted
I20240503 02:15:39.969483  2542 consensus_queue.cc:237] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:39.973098  4326 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.973539  4591 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.973960  8249 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.974157  2672 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:39.975261  8247 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.975844  8247 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:39.991509  8249 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:39.998793  4591 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:40.001178  4326 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:40.001554  2131 catalog_manager.cc:5484] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:40.022552  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de1d29dbea945eba58f8eadb019d766 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:40.049077  2668 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:40.062494  2542 consensus_queue.cc:237] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:40.088352  4591 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.090848  4460 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.091960  8364 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.093367  2281 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.093418  8363 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.100442  8363 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.143232  8371 tablet_replica.cc:330] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:40.143594  8371 raft_consensus.cc:2229] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:40.144115  8371 raft_consensus.cc:2258] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:40.145503  8371 ts_tablet_manager.cc:1875] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:40.151701  8364 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:40.157959  8371 ts_tablet_manager.cc:1888] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:40.158155  8371 log.cc:1196] T 2de1d29dbea945eba58f8eadb019d766 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/2de1d29dbea945eba58f8eadb019d766
I20240503 02:15:40.159482  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 2de1d29dbea945eba58f8eadb019d766 (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
I20240503 02:15:40.160092  4591 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:40.161481  2281 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:40.162173  2668 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:40.169443  2136 catalog_manager.cc:5484] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:40.175649  4461 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:40.180473  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 9253de3f798f4a208d566765ed2ba54d with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:40.186779  8398 tablet_replica.cc:330] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:40.187156  8398 raft_consensus.cc:2229] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:40.187563  8398 raft_consensus.cc:2258] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:40.188567  8398 ts_tablet_manager.cc:1875] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:40.192492  4326 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.192628  4591 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.193361  7956 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.194175  7956 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.200415  2540 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:40.201517  8149 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:40.203972  8398 ts_tablet_manager.cc:1888] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:40.204237  8398 log.cc:1196] T 9253de3f798f4a208d566765ed2ba54d P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/9253de3f798f4a208d566765ed2ba54d
I20240503 02:15:40.205518  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 9253de3f798f4a208d566765ed2ba54d (table rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]) successfully deleted
I20240503 02:15:40.222416  8149 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:40.223465  2540 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:40.249465  2136 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:40.254390  4591 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:40.271997  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eeb69e5eb3c459ca749aa37672ec057 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:40.273495  8398 tablet_replica.cc:330] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:40.273903  8398 raft_consensus.cc:2229] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:40.274403  8398 raft_consensus.cc:2258] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:40.275671  8398 ts_tablet_manager.cc:1875] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:40.289734  4327 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:40.305068  8398 ts_tablet_manager.cc:1888] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:40.305979  8398 log.cc:1196] T 1eeb69e5eb3c459ca749aa37672ec057 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/1eeb69e5eb3c459ca749aa37672ec057
I20240503 02:15:40.307682  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 1eeb69e5eb3c459ca749aa37672ec057 (table rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]) successfully deleted
I20240503 02:15:40.533556  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:40.554091  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:40.568850  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:40.604854  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:40.623347  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:40.689514  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:40.728082  2668 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:40.777670  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:40.781625  2281 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:40.782455  8149 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:40.811293  4194 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:40.815050  8149 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:40.827872  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:40.854689  8149 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:40.865067  2136 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:40.874830  4194 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:40.937726  2281 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:41.001515  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a3dcc2de594c8982477a0229f71191 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:41.032398  8497 tablet_replica.cc:330] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:41.033751  8497 raft_consensus.cc:2229] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:41.034948  8497 raft_consensus.cc:2258] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:41.036449  8497 ts_tablet_manager.cc:1875] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:41.088238  8497 ts_tablet_manager.cc:1888] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:41.088515  8497 log.cc:1196] T 62a3dcc2de594c8982477a0229f71191 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/62a3dcc2de594c8982477a0229f71191
I20240503 02:15:41.089802  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 62a3dcc2de594c8982477a0229f71191 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:15:41.324622  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:41.334136  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:41.390801  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:41.479213  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.505801  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:41.539057  2410 consensus_queue.cc:237] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:41.587052  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.599480  4327 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:41.599934  2669 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:41.600597  8432 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:41.601437  8432 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:41.609124  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:41.625553  4460 raft_consensus.cc:1261] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:41.628284  7962 consensus_queue.cc:1035] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:41.650876  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.694165  7962 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:41.699110  2668 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:41.708940  2134 catalog_manager.cc:5484] T f084f1783d4d477e9c3791984e681f4c P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:41.715124  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:41.714735  4460 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:41.717020  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.727764  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.746999  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:41.756363  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:41.764699  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:41.798638  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:41.834936  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.837430  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240503 02:15:41.926641  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 76332us
I20240503 02:15:41.927258  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.935388  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.939846  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet f084f1783d4d477e9c3791984e681f4c with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:41.942252  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:41.950459  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 58535.4us
I20240503 02:15:41.966764  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:41.971547  4326 raft_consensus.cc:2932] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:41.977730  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:41.980777  8644 tablet_replica.cc:330] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:41.991870  8644 raft_consensus.cc:2229] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:41.992319  8644 raft_consensus.cc:2258] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:41.993263  8644 ts_tablet_manager.cc:1875] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:42.016790  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.021893  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.153724  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:42.171224  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:42.177850  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.199482  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.213177  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.227475  8644 ts_tablet_manager.cc:1888] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:42.227787  8644 log.cc:1196] T f084f1783d4d477e9c3791984e681f4c P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/f084f1783d4d477e9c3791984e681f4c
I20240503 02:15:42.228928  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet f084f1783d4d477e9c3791984e681f4c (table rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]) successfully deleted
I20240503 02:15:42.232148  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:42.237979  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:42.254657  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:42.599668  2410 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:42.611549  4327 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.612911  8611 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.618336  4460 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.619992  2669 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.621274  8432 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.623050  7962 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.643496  8432 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:42.648986  2668 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:42.670006  4327 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:42.680557  2668 consensus_queue.cc:237] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:42.681890  2138 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:42.692425  4461 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:42.712998  4196 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.713052  4591 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.713954  8149 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.714886  8149 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.741408  2540 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:42.747999  8149 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:42.759305  8149 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.768692  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 0137e646f1bd452b950d870107afa060 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:42.799151  4196 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.805718  2280 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:42.815510  2668 consensus_queue.cc:237] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:42.819116  8766 tablet_replica.cc:330] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:42.818946  4591 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.819697  8766 raft_consensus.cc:2229] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:42.823109  8766 raft_consensus.cc:2258] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:42.824043  2138 catalog_manager.cc:5484] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.826730  8766 ts_tablet_manager.cc:1875] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:42.837837  2540 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.844519  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet a1949a5c7cc741b19da021eccb7d0dd1 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:42.857187  8766 ts_tablet_manager.cc:1888] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:42.863703  8766 log.cc:1196] T 0137e646f1bd452b950d870107afa060 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/0137e646f1bd452b950d870107afa060
I20240503 02:15:42.865967  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 0137e646f1bd452b950d870107afa060 (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
I20240503 02:15:42.886723  4593 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:42.887552  8740 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:42.912449  4592 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:42.912431  2279 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:42.913165  8642 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:42.913250  8741 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:42.925907  2540 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:42.928938  8642 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:42.943820  8741 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.956465  2131 catalog_manager.cc:5484] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.959810  8776 tablet_replica.cc:330] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:42.962965  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e61eb7d9e6041b6b99d5d5328943140 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:42.967963  8776 raft_consensus.cc:2229] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:42.969017  8776 raft_consensus.cc:2258] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:42.969942  4593 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.972386  2281 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.973213  8776 ts_tablet_manager.cc:1875] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:42.979956  8785 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:42.992818  8793 tablet_replica.cc:330] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:42.993064  8793 raft_consensus.cc:2229] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:42.993290  8793 raft_consensus.cc:2258] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:42.994073  8793 ts_tablet_manager.cc:1875] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:42.996380  2138 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:43.002295  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet d2351807a2ed47419f8d674fbeb9d0ec with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:43.005164  4592 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:43.007216  2540 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:43.013442  8800 tablet_replica.cc:330] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:43.015012  8800 raft_consensus.cc:2229] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:43.018616  8800 raft_consensus.cc:2258] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:43.023176  8800 ts_tablet_manager.cc:1875] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:43.027575  8793 ts_tablet_manager.cc:1888] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:43.027710  8793 log.cc:1196] T 0e61eb7d9e6041b6b99d5d5328943140 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e61eb7d9e6041b6b99d5d5328943140
I20240503 02:15:43.035743  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 0e61eb7d9e6041b6b99d5d5328943140 (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
I20240503 02:15:43.048709  8800 ts_tablet_manager.cc:1888] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:43.053071  8776 ts_tablet_manager.cc:1888] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:43.053270  8776 log.cc:1196] T a1949a5c7cc741b19da021eccb7d0dd1 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/a1949a5c7cc741b19da021eccb7d0dd1
I20240503 02:15:43.054334  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet a1949a5c7cc741b19da021eccb7d0dd1 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:15:43.092497  8800 log.cc:1196] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/d2351807a2ed47419f8d674fbeb9d0ec
I20240503 02:15:43.094084  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet d2351807a2ed47419f8d674fbeb9d0ec (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
I20240503 02:15:43.127746  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:43.192672  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:43.197117  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.258497  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:43.354105  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.373884  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.388912  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.404191  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:43.793242  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:43.838551  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.847589  2668 consensus_queue.cc:237] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:43.854557  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:43.876322  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.876649  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:43.957074  4593 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:43.958245  8149 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:43.973415  4461 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:43.974670  8740 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:44.010012  2542 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:44.010864  8149 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:44.029556  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:44.082087  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:44.103526  8149 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:44.126449  4593 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:44.147552  2131 catalog_manager.cc:5484] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:44.151229  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.152591  4461 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:44.157536  2542 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:44.195325  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 190ebb1335d64b279a41fa38feff22b4 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:44.206893  8996 tablet_replica.cc:330] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:44.208598  8996 raft_consensus.cc:2229] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:44.210578  8996 raft_consensus.cc:2258] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:44.211792  8996 ts_tablet_manager.cc:1875] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:44.234365  8996 ts_tablet_manager.cc:1888] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:44.235188  8996 log.cc:1196] T 190ebb1335d64b279a41fa38feff22b4 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/190ebb1335d64b279a41fa38feff22b4
I20240503 02:15:44.236498  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 190ebb1335d64b279a41fa38feff22b4 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:15:44.290796  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:44.345157  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:44.362653  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.387959  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:44.401690  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.416707  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.417884  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:44.421121  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:44.442541  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:44.455205  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.460942  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.544173  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:44.570452  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.592237  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.608908  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.610541  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:44.648012  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.650509  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:44.664868  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.674687  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:44.685933  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:44.700851  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:44.752254  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:44.802383  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:45.165583  2410 consensus_queue.cc:237] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:45.196516  4593 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:45.198691  2279 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:45.199695  7962 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:45.202080  7962 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:45.229813  8432 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.238478  2279 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.245754  4593 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.263017  2138 catalog_manager.cc:5484] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.366664  2668 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:45.369829  4593 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:45.370312  4327 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:45.371096  9079 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:45.372841  9079 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:45.377931  9079 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.381573  4593 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.388342  4327 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.448444  2132 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:45.462980  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet a7ab8b0c8db54565a6066b9a7d1569d8 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:45.465274  2521 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eeb69e5eb3c459ca749aa37672ec057 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:45.525163  9157 tablet_replica.cc:330] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:45.525717  9157 raft_consensus.cc:2229] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:45.526130  9157 raft_consensus.cc:2258] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:45.527673  9157 ts_tablet_manager.cc:1875] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:45.541150  9157 ts_tablet_manager.cc:1888] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:45.543442  9157 log.cc:1196] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7ab8b0c8db54565a6066b9a7d1569d8
I20240503 02:15:45.545982  9157 tablet_replica.cc:330] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:45.548229  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet a7ab8b0c8db54565a6066b9a7d1569d8 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
I20240503 02:15:45.548858  9157 raft_consensus.cc:2229] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:45.549239  9157 raft_consensus.cc:2258] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:45.550375  9157 ts_tablet_manager.cc:1875] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:45.568056  9157 ts_tablet_manager.cc:1888] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:45.568835  9157 log.cc:1196] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/1eeb69e5eb3c459ca749aa37672ec057
I20240503 02:15:45.570185  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 1eeb69e5eb3c459ca749aa37672ec057 (table rebalance_test_table_1 [id=4d368e5356e14979916b32fbd558c44a]) successfully deleted
I20240503 02:15:45.673077  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:45.713119  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:45.720090  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:45.884358  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:45.908805  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:45.966408  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:45.966672  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:46.065196  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.568962  2542 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:46.586735  4326 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:46.590023  9336 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:46.632100  2279 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:46.663736  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:46.679667  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:46.687336  9336 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:46.740945  9336 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:46.748698  2139 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:46.755358  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:46.809772  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.813578  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.814190  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.822703  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.822762  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:46.838927  2279 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:46.841621  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.869477  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:46.874758  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:46.890149  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.909713  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.910823  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:46.921768  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:46.952328  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:46.957154  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:46.957391  4326 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:46.968619  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.007052  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:47.009547  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.083793  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:47.101236  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.105520  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:47.121774  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.150460  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.151264  9440 tablet_replica.cc:330] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:47.151479  9440 raft_consensus.cc:2229] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:47.151809  9440 raft_consensus.cc:2258] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:47.152624  9440 ts_tablet_manager.cc:1875] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:47.154525  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.184274  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.204262  9440 ts_tablet_manager.cc:1888] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:47.222821  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:47.233999  9440 log.cc:1196] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/51dbf0fd54b14c2fa21d3e62104c9b87
I20240503 02:15:47.251211  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 51dbf0fd54b14c2fa21d3e62104c9b87 (table rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]) successfully deleted
I20240503 02:15:47.251310  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:47.251801  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:47.253433  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:47.298223  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:47.305119  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:47.861065  2279 consensus_queue.cc:237] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:47.874408  2281 consensus_queue.cc:237] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:47.880450  4326 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.881265  9070 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:47.884071  4196 raft_consensus.cc:1261] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.884716  9070 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:47.888785  2280 consensus_queue.cc:237] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:47.891942  2668 raft_consensus.cc:1261] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.893958  2410 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.895568  9530 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:47.901849  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:47.914359  4194 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.918267  2668 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:47.940155  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:47.954777  9070 consensus_queue.cc:1035] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:15:47.957123  9070 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:47.992434  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:48.049475  9552 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:15:48.061622  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 139384us
I20240503 02:15:48.062546  9070 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:48.063392  9070 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.072664  2669 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:48.083568  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:48.096962  2131 catalog_manager.cc:5484] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:48.124114  4326 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:48.126930  4194 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.127974  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:48.147593  2132 catalog_manager.cc:5484] T f0a088d230d0462980648628eae548d2 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.163982  9557 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.179069  2410 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.189384  2669 raft_consensus.cc:2932] T f0a088d230d0462980648628eae548d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.207305  4194 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.232038  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:48.230319  2140 catalog_manager.cc:5484] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:48.257167  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:48.282565  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:48.288956  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 586ebf467f0f427fa62dee6223a39848 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:48.296151  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet f0a088d230d0462980648628eae548d2 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:48.299561  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 734027e195414b15811ac9d32b132db8 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:48.322970  9586 tablet_replica.cc:330] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:48.323526  9586 raft_consensus.cc:2229] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:48.324697  9586 raft_consensus.cc:2258] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:48.335105  9586 ts_tablet_manager.cc:1875] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:48.365140  9587 tablet_replica.cc:330] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:48.375949  9587 raft_consensus.cc:2229] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:48.385002  9588 tablet_replica.cc:330] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:48.385252  9588 raft_consensus.cc:2229] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:48.386003  9586 ts_tablet_manager.cc:1888] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:48.386158  9586 log.cc:1196] T 586ebf467f0f427fa62dee6223a39848 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/586ebf467f0f427fa62dee6223a39848
I20240503 02:15:48.387123  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 586ebf467f0f427fa62dee6223a39848 (table rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]) successfully deleted
I20240503 02:15:48.387193  9587 raft_consensus.cc:2258] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:48.394878  9588 raft_consensus.cc:2258] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:48.395234  9587 ts_tablet_manager.cc:1875] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:48.398743  9588 ts_tablet_manager.cc:1875] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:48.496724  9588 ts_tablet_manager.cc:1888] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:48.497046  9588 log.cc:1196] T 734027e195414b15811ac9d32b132db8 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/734027e195414b15811ac9d32b132db8
I20240503 02:15:48.461323  9587 ts_tablet_manager.cc:1888] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:48.497668  9587 log.cc:1196] T f0a088d230d0462980648628eae548d2 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0a088d230d0462980648628eae548d2
I20240503 02:15:48.498757  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet f0a088d230d0462980648628eae548d2 (table rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]) successfully deleted
I20240503 02:15:48.505071  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 734027e195414b15811ac9d32b132db8 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:15:48.828378  2281 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:48.855715  4459 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:48.856633  9552 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:48.879235  9552 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:48.882174  4461 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:48.897500  2140 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:48.945150  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:48.979440  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:49.019779  2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 02e0b954008c4f30af414da6cb5d59c2 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:15:49.022054  2281 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:49.025447  4461 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:49.026473  9549 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:15:49.029214  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e0b954008c4f30af414da6cb5d59c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:49.031491  9549 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:49.046237  4459 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:49.049425  9718 tablet_replica.cc:330] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:15:49.049821  9718 raft_consensus.cc:2229] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:49.050199  9718 raft_consensus.cc:2258] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:49.051465  9718 ts_tablet_manager.cc:1875] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:49.056098  2136 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:49.059499  2127 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 02e0b954008c4f30af414da6cb5d59c2 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:49.072970  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.087291  9718 ts_tablet_manager.cc:1888] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:49.134624  9718 log.cc:1196] T 02e0b954008c4f30af414da6cb5d59c2 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/02e0b954008c4f30af414da6cb5d59c2
I20240503 02:15:49.137358  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 02e0b954008c4f30af414da6cb5d59c2 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:15:49.182426  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.245807  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:49.261493  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:49.272563  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.285198  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.348341  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:49.350870  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.367058  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.401409  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:49.438900  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.469923  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.473556  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.502876  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:49.518977  9784 ts_tablet_manager.cc:902] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:49.540109  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:49.558068  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.568324  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:49.593325  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:49.611531  9784 tablet_copy_client.cc:308] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:49.611701  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:49.612327  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:49069)
I20240503 02:15:49.612473  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:49069: session id = 7fd07044862a481aa387698017467c01-02e0b954008c4f30af414da6cb5d59c2
I20240503 02:15:49.616050  2301 tablet_copy_source_session.cc:205] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:49.619781  9784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:49.639091  9784 tablet_copy_client.cc:783] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:49.639555  9784 tablet_copy_client.cc:647] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:49.643497  9784 tablet_copy_client.cc:523] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:49.647811  9784 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:49.668041  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.675175  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:49.687570  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:49.729064  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:49.743647  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.774789  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.795780  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.806160  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.841133  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:49.841274  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:49.879225  9784 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:49.880977  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:49.883787  9784 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:49.884166  9784 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.236s	user 0.044s	sys 0.023s
I20240503 02:15:49.885075  9784 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:49.885309  9784 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:49.896706  9784 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:49.939159  9784 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.055s	user 0.001s	sys 0.003s
I20240503 02:15:49.942363  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 7fd07044862a481aa387698017467c01-02e0b954008c4f30af414da6cb5d59c2 received from {username='slave'} at 127.20.218.72:49069
I20240503 02:15:49.942512  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 7fd07044862a481aa387698017467c01-02e0b954008c4f30af414da6cb5d59c2 on tablet 02e0b954008c4f30af414da6cb5d59c2 with peer 7fd07044862a481aa387698017467c01
W20240503 02:15:50.171295  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 107831us
I20240503 02:15:50.400053  4592 raft_consensus.cc:1203] T 02e0b954008c4f30af414da6cb5d59c2 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:15:50.412961  2669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191"
dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55482
I20240503 02:15:50.413398  2669 raft_consensus.cc:595] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:15:50.491142  2540 consensus_queue.cc:237] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:50.501929  9824 raft_consensus.cc:979] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: : Instructing follower 08055479ea104aa290d19955d3720850 to start an election
I20240503 02:15:50.502111  9824 raft_consensus.cc:1067] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Signalling peer 08055479ea104aa290d19955d3720850 to start an election
I20240503 02:15:50.503818  2279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191"
dest_uuid: "08055479ea104aa290d19955d3720850"
 from {username='slave'} at 127.20.218.68:50175
I20240503 02:15:50.504176  2279 raft_consensus.cc:482] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:15:50.504313  2279 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:50.508388  2279 raft_consensus.cc:504] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:50.510316  2669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
I20240503 02:15:50.510602  2669 raft_consensus.cc:3032] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:15:50.510681  2669 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:15:50.511073  2669 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:50.511530  2669 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:50.509922  2279 leader_election.cc:288] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 2 election: Requested vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:15:50.514065  4196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "08055479ea104aa290d19955d3720850" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1d36807868fe4a82bca42c25831ce8a2"
I20240503 02:15:50.514509  4196 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:50.531052  4196 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 2.
I20240503 02:15:50.531630  2669 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08055479ea104aa290d19955d3720850 in term 2.
I20240503 02:15:50.532321  2216 leader_election.cc:302] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2; no voters: 
I20240503 02:15:50.532747  9552 raft_consensus.cc:2781] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:50.533192  9552 raft_consensus.cc:686] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Becoming Leader. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:50.533628  9552 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:50.537940  2136 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 reported cstate change: term changed from 1 to 2, leader changed from e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) to 08055479ea104aa290d19955d3720850 (127.20.218.65). New cstate: current_term: 2 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:50.561407  4196 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:50.563700  9560 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:50.569386  2410 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:50.581930  9560 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:50.586302  4459 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:50.587170  9431 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:50.590793  9837 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.591579  4459 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.592808  4194 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:50.594151  2540 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:50.599889  9431 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:50.601809  2136 catalog_manager.cc:5484] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:50.603358  2410 consensus_queue.cc:237] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:50.610296  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet ae239d73f0bc4d9aac025e40489e81a6 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:50.611236  2542 consensus_queue.cc:237] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
W20240503 02:15:50.615176  2475 consensus_peers.cc:485] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae239d73f0bc4d9aac025e40489e81a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:50.617105  9932 tablet_replica.cc:330] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:50.623090  9932 raft_consensus.cc:2229] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:50.623509  9932 raft_consensus.cc:2258] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:50.624447  9932 ts_tablet_manager.cc:1875] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:50.634480  2279 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:50.636476  4461 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:50.637032  4196 raft_consensus.cc:1261] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:50.639703  2139 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.650476  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 8870feaf47a64a929dd2e584ece47218 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:50.656845  9931 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:50.659266  9932 ts_tablet_manager.cc:1888] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:50.659425  9932 log.cc:1196] T ae239d73f0bc4d9aac025e40489e81a6 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae239d73f0bc4d9aac025e40489e81a6
I20240503 02:15:50.659301  9431 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:50.660570  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet ae239d73f0bc4d9aac025e40489e81a6 (table rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]) successfully deleted
I20240503 02:15:50.672660  9900 consensus_queue.cc:1035] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:50.677603  9900 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:50.682245  9931 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.693580  2279 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.693570  4459 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.703527  2139 catalog_manager.cc:5484] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 4 to 5, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.724478  4196 raft_consensus.cc:2932] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:50.732183  2540 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:50.739214  2136 catalog_manager.cc:5484] T ae239d73f0bc4d9aac025e40489e81a6 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 6 to 7, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:50.744379  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ae239d73f0bc4d9aac025e40489e81a6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:50.746500  9941 tablet_replica.cc:330] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:50.746876  9941 raft_consensus.cc:2229] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:50.747323  9941 raft_consensus.cc:2258] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:50.748155  9941 ts_tablet_manager.cc:1875] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:50.750494  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7c9c29384949fb95cb0c3b765f57b0 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:50.756170  9958 tablet_replica.cc:330] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:50.756726  9958 raft_consensus.cc:2229] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:50.757583  9958 raft_consensus.cc:2258] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:50.759189  9958 ts_tablet_manager.cc:1875] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:50.785318  9941 ts_tablet_manager.cc:1888] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:50.795987  9941 log.cc:1196] T 8870feaf47a64a929dd2e584ece47218 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/8870feaf47a64a929dd2e584ece47218
I20240503 02:15:50.797484  9958 ts_tablet_manager.cc:1888] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:50.797708  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 8870feaf47a64a929dd2e584ece47218 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:15:50.818531  9958 log.cc:1196] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f7c9c29384949fb95cb0c3b765f57b0
I20240503 02:15:50.826395  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 9f7c9c29384949fb95cb0c3b765f57b0 (table rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]) successfully deleted
I20240503 02:15:50.926858  4196 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:50.928802  9552 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:51.024979  2669 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:51.100303  9902 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:51.117412 10017 ts_tablet_manager.cc:902] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:51.147675  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.162138  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:51.179701  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.196641 10017 tablet_copy_client.cc:308] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:51.198272  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet ae239d73f0bc4d9aac025e40489e81a6 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:43165)
I20240503 02:15:51.198441  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet ae239d73f0bc4d9aac025e40489e81a6 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:43165: session id = 9e699d9f984b4cf2b8b468322759b78c-ae239d73f0bc4d9aac025e40489e81a6
I20240503 02:15:51.201880  2562 tablet_copy_source_session.cc:205] T ae239d73f0bc4d9aac025e40489e81a6 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:51.207798 10017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae239d73f0bc4d9aac025e40489e81a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:51.224363  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.231411  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:51.235661 10017 tablet_copy_client.cc:783] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:51.236686 10017 tablet_copy_client.cc:647] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:51.239593 10017 tablet_copy_client.cc:523] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:51.253574 10017 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
W20240503 02:15:51.288995  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 133458us
I20240503 02:15:51.315395  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:51.355618  2260 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:51.395313  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.509863 10017 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:51.510601 10017 tablet_bootstrap.cc:492] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:51.511014 10017 ts_tablet_manager.cc:1367] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.268s	user 0.059s	sys 0.006s
I20240503 02:15:51.512063 10017 raft_consensus.cc:348] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:51.512342 10017 raft_consensus.cc:729] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:51.512888 10017 consensus_queue.cc:260] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:51.588204 10017 ts_tablet_manager.cc:1398] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.077s	user 0.013s	sys 0.028s
I20240503 02:15:51.589036  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-ae239d73f0bc4d9aac025e40489e81a6 received from {username='slave'} at 127.20.218.70:43165
I20240503 02:15:51.589182  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-ae239d73f0bc4d9aac025e40489e81a6 on tablet ae239d73f0bc4d9aac025e40489e81a6 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:51.591724  2260 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:51.608386  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:51.615763  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.621387  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.662945  4327 raft_consensus.cc:1203] T ae239d73f0bc4d9aac025e40489e81a6 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:15:51.691376  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:51.695394  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.704325  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:51.718780  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:51.807014  2279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec"
dest_uuid: "08055479ea104aa290d19955d3720850"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54094
I20240503 02:15:51.807402  2279 raft_consensus.cc:595] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:15:51.910218  9552 raft_consensus.cc:979] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: : Instructing follower 08bef4c144bc4e4d8cb5836aee8da852 to start an election
I20240503 02:15:51.910399  9552 raft_consensus.cc:1067] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Signalling peer 08bef4c144bc4e4d8cb5836aee8da852 to start an election
I20240503 02:15:51.912679  2540 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec"
dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
 from {username='slave'} at 127.20.218.65:46881
I20240503 02:15:51.913029  2540 raft_consensus.cc:482] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:15:51.913180  2540 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:51.925411  2540 raft_consensus.cc:504] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:51.928592  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:51.928894  2279 raft_consensus.cc:3032] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:15:51.928973  2279 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:51.929404  2279 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:51.929894  2279 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:51.934218  2279 raft_consensus.cc:2454] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 2.
I20240503 02:15:51.935101  2478 leader_election.cc:302] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852; no voters: 
I20240503 02:15:51.935498  9950 raft_consensus.cc:2781] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:51.935662  9950 raft_consensus.cc:686] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:15:51.936050  2540 leader_election.cc:288] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 2 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 7fd07044862a481aa387698017467c01 (127.20.218.72:37033)
I20240503 02:15:51.936009  9950 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:51.938233  4592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2351807a2ed47419f8d674fbeb9d0ec" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7fd07044862a481aa387698017467c01"
I20240503 02:15:51.938640  4592 raft_consensus.cc:3037] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:51.954787  4592 raft_consensus.cc:2454] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 2.
I20240503 02:15:51.959398  2136 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 1 to 2, leader changed from 08055479ea104aa290d19955d3720850 (127.20.218.65) to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:52.154423  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.182693  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:15:52.242131  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 45057.5us
I20240503 02:15:52.264925  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.391417  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.411854  2279 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:52.412108  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.412675 10114 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:52.440455  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.459740  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.505717  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.525398  4592 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:52.526216 10114 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:52.529726  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.576663  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:52.575327  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 80786us
I20240503 02:15:52.608793  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.651893  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:52.745103  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:52.759673  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.773232  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.796454  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:52.819056  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.822981  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:52.843694  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.844779  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.862627  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.884302  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:52.900410  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.939617  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:52.959462  4307 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.959647  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:52.962095  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:52.966354  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:53.021915  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 63062.1us
I20240503 02:15:53.024314  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:53.032402  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:53.058847  2260 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:53.078598  2391 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:15:53.326212  2281 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:53.329144  4195 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:15:53.329931  9902 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:53.348912  9552 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.354774  2134 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 7, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 2 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.359952  4195 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.454336  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a3dcc2de594c8982477a0229f71191 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:53.453231  2281 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
W20240503 02:15:53.458671  2214 consensus_peers.cc:485] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a3dcc2de594c8982477a0229f71191. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:53.459897  4195 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:15:53.460705  9552 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:53.465104  9552 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.465839  2281 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15"
dest_uuid: "08055479ea104aa290d19955d3720850"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54176
I20240503 02:15:53.466192  2281 raft_consensus.cc:595] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:15:53.467173  4195 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.477684  2127 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 62a3dcc2de594c8982477a0229f71191 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:53.480163  2134 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 7 to 8, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 2 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.498864 10307 tablet_replica.cc:330] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:53.502249 10307 raft_consensus.cc:2229] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:53.503163 10307 raft_consensus.cc:2258] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:53.505391 10307 ts_tablet_manager.cc:1875] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:53.517531  9902 raft_consensus.cc:979] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: : Instructing follower 6d54feb0bbc14c109783371d2939b234 to start an election
I20240503 02:15:53.517791  9902 raft_consensus.cc:1067] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Signalling peer 6d54feb0bbc14c109783371d2939b234 to start an election
I20240503 02:15:53.518957  2411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15"
dest_uuid: "6d54feb0bbc14c109783371d2939b234"
 from {username='slave'} at 127.20.218.65:33571
I20240503 02:15:53.519325  2411 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:15:53.519541  2411 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:53.524121  2411 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:53.526036  2541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:15:53.526306  2541 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:53.526795  4460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "408af0ab5520465e942cc714bfc6c73b"
I20240503 02:15:53.527107  4460 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:53.529253  2541 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 2.
I20240503 02:15:53.529896 10307 ts_tablet_manager.cc:1888] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:15:53.530077 10307 log.cc:1196] T 62a3dcc2de594c8982477a0229f71191 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/62a3dcc2de594c8982477a0229f71191
I20240503 02:15:53.530589  4460 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 2.
I20240503 02:15:53.531143  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 62a3dcc2de594c8982477a0229f71191 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:15:53.531585  2348 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 408af0ab5520465e942cc714bfc6c73b, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:15:53.531970 10257 raft_consensus.cc:2781] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:15:53.532145 10257 raft_consensus.cc:686] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 2 LEADER]: Becoming Leader. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:15:53.532611 10257 consensus_queue.cc:237] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:53.532972  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9e699d9f984b4cf2b8b468322759b78c"
I20240503 02:15:53.533385  4326 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:53.534605  2411 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [CANDIDATE]: Term 2 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:15:53.536787  4326 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 2.
I20240503 02:15:53.548599  2134 catalog_manager.cc:5484] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: term changed from 1 to 2, leader changed from 08055479ea104aa290d19955d3720850 (127.20.218.65) to 6d54feb0bbc14c109783371d2939b234 (127.20.218.66). New cstate: current_term: 2 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.549899  2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "6d54feb0bbc14c109783371d2939b234" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:53.550170  2281 raft_consensus.cc:3032] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:15:53.550272  2281 raft_consensus.cc:729] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:53.550606  2281 consensus_queue.cc:260] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:53.551268  2281 raft_consensus.cc:3037] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:15:53.555413  2281 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d54feb0bbc14c109783371d2939b234 in term 2.
I20240503 02:15:53.696899  2412 consensus_queue.cc:237] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:53.700999  2541 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:53.701684  9949 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:53.702332  4326 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:15:53.703635  2541 consensus_queue.cc:237] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:53.706140  9949 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:15:53.707746  4327 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:53.708616 10241 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:53.709154  9949 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.715797  2540 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.722746  2668 raft_consensus.cc:1261] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:53.723045 10241 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.723799 10114 consensus_queue.cc:1035] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:53.727051  4327 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.734184  2134 catalog_manager.cc:5484] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.748019  2135 catalog_manager.cc:5484] T 2de1d29dbea945eba58f8eadb019d766 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.762759  4573 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de1d29dbea945eba58f8eadb019d766 with delete_type TABLET_DATA_TOMBSTONED (TS 7fd07044862a481aa387698017467c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34002
I20240503 02:15:53.763458  4326 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.772157  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet eadcb49d12664e6fb5a5ad8dbe47f342 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:53.772557 10307 tablet_replica.cc:330] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:53.772744 10307 raft_consensus.cc:2229] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:53.772965 10307 raft_consensus.cc:2258] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:53.773717 10307 ts_tablet_manager.cc:1875] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:53.782917 10307 ts_tablet_manager.cc:1888] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 02:15:53.783093 10307 log.cc:1196] T eadcb49d12664e6fb5a5ad8dbe47f342 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/eadcb49d12664e6fb5a5ad8dbe47f342
I20240503 02:15:53.784222  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet eadcb49d12664e6fb5a5ad8dbe47f342 (table rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]) successfully deleted
I20240503 02:15:53.787348 10379 tablet_replica.cc:330] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: stopping tablet replica
I20240503 02:15:53.787631 10379 raft_consensus.cc:2229] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:53.787961 10379 raft_consensus.cc:2258] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:53.788779 10379 ts_tablet_manager.cc:1875] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:53.810475  2668 raft_consensus.cc:2932] T 2de1d29dbea945eba58f8eadb019d766 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:53.817530 10379 ts_tablet_manager.cc:1888] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:53.817930 10379 log.cc:1196] T 2de1d29dbea945eba58f8eadb019d766 P 7fd07044862a481aa387698017467c01: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/wals/2de1d29dbea945eba58f8eadb019d766
I20240503 02:15:53.819269  2126 catalog_manager.cc:4830] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): tablet 2de1d29dbea945eba58f8eadb019d766 (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
I20240503 02:15:53.819401  2540 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:53.842972  4591 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:15:53.844064 10114 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:53.847641 10241 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:53.848345  4591 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:53.868180  2134 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:53.879484  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet d2351807a2ed47419f8d674fbeb9d0ec with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:53.879992  2540 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
W20240503 02:15:53.884706  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2351807a2ed47419f8d674fbeb9d0ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:53.889797  4591 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:15:53.891047 10241 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:15:53.896787 10241 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.902674  4592 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.907255 10398 tablet_replica.cc:330] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:53.908196 10398 raft_consensus.cc:2229] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:53.911085  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d2351807a2ed47419f8d674fbeb9d0ec with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:53.913566  2134 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 7 to 8, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:53.922835 10398 raft_consensus.cc:2258] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:53.961670 10396 ts_tablet_manager.cc:902] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:15:54.006521 10398 ts_tablet_manager.cc:1875] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:54.023825 10396 tablet_copy_client.cc:308] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:15:54.025406  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 62a3dcc2de594c8982477a0229f71191 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:34139)
I20240503 02:15:54.025645  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 62a3dcc2de594c8982477a0229f71191 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:34139: session id = 9e699d9f984b4cf2b8b468322759b78c-62a3dcc2de594c8982477a0229f71191
I20240503 02:15:54.030069  2301 tablet_copy_source_session.cc:205] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:54.032469 10398 ts_tablet_manager.cc:1888] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:15:54.032778 10398 log.cc:1196] T d2351807a2ed47419f8d674fbeb9d0ec P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/d2351807a2ed47419f8d674fbeb9d0ec
I20240503 02:15:54.033236 10396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:54.034197  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet d2351807a2ed47419f8d674fbeb9d0ec (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
I20240503 02:15:54.050346 10396 tablet_copy_client.cc:783] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:54.050817 10396 tablet_copy_client.cc:647] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:54.053036 10396 tablet_copy_client.cc:523] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:54.065069 10396 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:15:54.069144  2540 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:54.070056 10257 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:54.078627  2281 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:54.079756  9949 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:54.083523  4326 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:54.086040  9949 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:54.090483  4460 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:15:54.092525 10257 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:54.300904  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:54.308036  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:54.323083  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:54.360525 10396 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:54.362365 10396 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:15:54.362758 10480 ts_tablet_manager.cc:902] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:15:54.364904 10396 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.300s	user 0.051s	sys 0.020s
I20240503 02:15:54.369351 10396 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:54.369686 10396 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:15:54.369947 10396 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:54.370589 10396 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240503 02:15:54.378412  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-62a3dcc2de594c8982477a0229f71191 received from {username='slave'} at 127.20.218.70:34139
I20240503 02:15:54.378579  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-62a3dcc2de594c8982477a0229f71191 on tablet 62a3dcc2de594c8982477a0229f71191 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:54.382906  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.391585  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:54.401427  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.410125  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.474521 10480 tablet_copy_client.cc:308] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:15:54.475929  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet d2351807a2ed47419f8d674fbeb9d0ec from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:57071)
I20240503 02:15:54.476157  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet d2351807a2ed47419f8d674fbeb9d0ec from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:57071: session id = 1d36807868fe4a82bca42c25831ce8a2-d2351807a2ed47419f8d674fbeb9d0ec
I20240503 02:15:54.479977  2562 tablet_copy_source_session.cc:205] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:54.484126 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:54.489025  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.572790  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.581084  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:54.597759  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:54.600072 10480 tablet_copy_client.cc:783] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:54.600481 10480 tablet_copy_client.cc:647] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:54.602583 10480 tablet_copy_client.cc:523] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:54.626813  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:54.653847 10480 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:15:54.659144  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.706318  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.709807  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:54.726112  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:54.829911  4326 raft_consensus.cc:1203] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:15:54.931304 10480 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:54.931888 10480 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:15:54.932209 10480 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.278s	user 0.048s	sys 0.010s
I20240503 02:15:54.933127 10480 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:54.933401 10480 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:15:54.933709 10480 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:54.941085 10480 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.008s
I20240503 02:15:54.944203  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-d2351807a2ed47419f8d674fbeb9d0ec received from {username='slave'} at 127.20.218.69:57071
I20240503 02:15:54.944384  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-d2351807a2ed47419f8d674fbeb9d0ec on tablet d2351807a2ed47419f8d674fbeb9d0ec with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:55.043536  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:55.075822  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:55.086601  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:55.150278  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:55.158644  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:55.166518  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:55.167841  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:55.195472  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:55.356369  4195 raft_consensus.cc:1203] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:15:55.371206  2412 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:55.374547  4326 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:55.377429 10374 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.383517  2668 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:55.383565  4592 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:55.386904 10374 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.387806 10374 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.388900  4196 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:55.389809 10257 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.402101 10374 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.405050  4196 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.405202  4592 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.412616  2134 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.446054  4326 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.466579  2669 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:15:55.489674  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b822c691d824b2ba25e967535c9734e with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:55.518736 10670 tablet_replica.cc:330] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:55.520551 10670 raft_consensus.cc:2229] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:55.526031 10670 raft_consensus.cc:2258] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:55.530982 10670 ts_tablet_manager.cc:1875] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:55.565315 10670 ts_tablet_manager.cc:1888] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:15:55.586066 10670 log.cc:1196] T 9b822c691d824b2ba25e967535c9734e P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b822c691d824b2ba25e967535c9734e
I20240503 02:15:55.588650  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 9b822c691d824b2ba25e967535c9734e (table rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]) successfully deleted
I20240503 02:15:55.597045  2412 consensus_queue.cc:237] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:55.611145  2540 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:15:55.611294  4326 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:15:55.611863  4460 raft_consensus.cc:1261] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:15:55.612087 10645 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.612706 10645 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.614420 10645 consensus_queue.cc:1035] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:55.640743 10645 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 6d54feb0bbc14c109783371d2939b234 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:55.644270  4459 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:55.658223  2138 catalog_manager.cc:5484] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:55.659343  4327 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:55.665622  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ba33132e784f408f3e26882aea3d15 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38192
I20240503 02:15:55.688511  2540 raft_consensus.cc:2932] T 75ba33132e784f408f3e26882aea3d15 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:55.719476 10709 tablet_replica.cc:330] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:15:55.719892 10709 raft_consensus.cc:2229] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:55.720247 10709 raft_consensus.cc:2258] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:55.721427 10709 ts_tablet_manager.cc:1875] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:15:55.724520  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 46041.8us
I20240503 02:15:55.752084 10709 ts_tablet_manager.cc:1888] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:15:55.753485 10709 log.cc:1196] T 75ba33132e784f408f3e26882aea3d15 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/75ba33132e784f408f3e26882aea3d15
I20240503 02:15:55.755409  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 75ba33132e784f408f3e26882aea3d15 (table rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]) successfully deleted
I20240503 02:15:56.226117  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:56.308372  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:56.368678  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:56.401984  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.443903  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.474802  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.488901  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:56.527408  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.686077  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:56.782943  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.812541  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:56.815796  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.883915  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.894546  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:56.952045  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:56.967635  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:56.974268  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:57.032867  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:57.105023  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.105190  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.188037  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:57.230013  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.262084  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.264302  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:57.271821  2412 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:15:57.288235  4460 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:57.288940 10374 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:57.292795 10374 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:57.297472  4460 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:57.302464  2138 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:15:57.413543  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 8870feaf47a64a929dd2e584ece47218 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47402
I20240503 02:15:57.414145 10984 tablet_replica.cc:330] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:15:57.414767 10984 raft_consensus.cc:2229] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:57.414749  2412 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:57.415328 10984 raft_consensus.cc:2258] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 02:15:57.418784  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8870feaf47a64a929dd2e584ece47218. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:15:57.416425 10984 ts_tablet_manager.cc:1875] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:57.419417  4460 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:15:57.420274 10645 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:15:57.428382 10984 ts_tablet_manager.cc:1888] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:57.428663 10984 log.cc:1196] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/8870feaf47a64a929dd2e584ece47218
I20240503 02:15:57.429822  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 8870feaf47a64a929dd2e584ece47218 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:15:57.430172 10645 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:57.439030  4460 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:15:57.463567  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8870feaf47a64a929dd2e584ece47218 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:15:57.468679  2136 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
W20240503 02:15:57.665699  2609 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 69475.9us
I20240503 02:15:57.665904  2540 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
W20240503 02:15:57.669904  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 50449.2us
I20240503 02:15:57.680418  4176 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.693086  2260 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:15:57.772550  4327 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:57.773129  4592 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:57.774073 10980 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:57.777195 10980 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:57.782536  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:57.800689  2412 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:15:57.801398 10767 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:15:57.803671 10980 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:57.804473  4326 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:57.817231  4592 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:57.829655  2136 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:57.834192  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 592cf4d5b35e49bfa4173343d64367f4 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36984
I20240503 02:15:57.868979 11063 tablet_replica.cc:330] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:15:57.871491  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.874753 11063 raft_consensus.cc:2229] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:15:57.875133 11063 raft_consensus.cc:2258] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:15:57.875998 11063 ts_tablet_manager.cc:1875] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:15:57.886612  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.891199  2411 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:57.895347  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:57.910360  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:57.933923 11063 ts_tablet_manager.cc:1888] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:15:57.935204 11063 log.cc:1196] T 592cf4d5b35e49bfa4173343d64367f4 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/592cf4d5b35e49bfa4173343d64367f4
I20240503 02:15:57.936199  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:57.936396  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 592cf4d5b35e49bfa4173343d64367f4 (table rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]) successfully deleted
I20240503 02:15:58.046423 11088 ts_tablet_manager.cc:902] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:15:58.083014 11088 tablet_copy_client.cc:308] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:15:58.084661  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 8870feaf47a64a929dd2e584ece47218 from peer 7fd07044862a481aa387698017467c01 ({username='slave'} at 127.20.218.72:35157)
I20240503 02:15:58.084846  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 8870feaf47a64a929dd2e584ece47218 from peer 7fd07044862a481aa387698017467c01 at {username='slave'} at 127.20.218.72:35157: session id = 7fd07044862a481aa387698017467c01-8870feaf47a64a929dd2e584ece47218
I20240503 02:15:58.087750  2432 tablet_copy_source_session.cc:205] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:15:58.090013 11088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:15:58.115461 11088 tablet_copy_client.cc:783] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 0 data blocks...
I20240503 02:15:58.115885 11088 tablet_copy_client.cc:647] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: tablet copy: Starting download of 1 WAL segments...
I20240503 02:15:58.118645 11088 tablet_copy_client.cc:523] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:15:58.130999 11088 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Bootstrap starting.
I20240503 02:15:58.261004  2279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191"
dest_uuid: "08055479ea104aa290d19955d3720850"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54370
I20240503 02:15:58.261262  2279 raft_consensus.cc:595] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Received request to transfer leadership
I20240503 02:15:58.312199 11088 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:15:58.312883 11088 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Bootstrap complete.
I20240503 02:15:58.313243 11088 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Time spent bootstrapping tablet: real 0.182s	user 0.079s	sys 0.023s
I20240503 02:15:58.314347 11088 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:58.314694 11088 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd07044862a481aa387698017467c01, State: Initialized, Role: FOLLOWER
I20240503 02:15:58.315006 11088 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:58.315718 11088 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:15:58.319219  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 7fd07044862a481aa387698017467c01-8870feaf47a64a929dd2e584ece47218 received from {username='slave'} at 127.20.218.72:35157
I20240503 02:15:58.319360  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 7fd07044862a481aa387698017467c01-8870feaf47a64a929dd2e584ece47218 on tablet 8870feaf47a64a929dd2e584ece47218 with peer 7fd07044862a481aa387698017467c01
I20240503 02:15:58.387869 10557 raft_consensus.cc:979] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: : Instructing follower 1d36807868fe4a82bca42c25831ce8a2 to start an election
I20240503 02:15:58.388095 10557 raft_consensus.cc:1067] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Signalling peer 1d36807868fe4a82bca42c25831ce8a2 to start an election
I20240503 02:15:58.390667  4196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191"
dest_uuid: "1d36807868fe4a82bca42c25831ce8a2"
 from {username='slave'} at 127.20.218.65:34381
I20240503 02:15:58.391075  4196 raft_consensus.cc:482] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:15:58.391227  4196 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:15:58.402065  4196 raft_consensus.cc:504] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:58.405961  2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:15:58.406255  2279 raft_consensus.cc:3032] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 02:15:58.406322  2279 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:15:58.406587  2279 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:58.406934  2279 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:15:58.411185  2279 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 3.
I20240503 02:15:58.412083  4132 leader_election.cc:302] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2; no voters: 
I20240503 02:15:58.413046  4196 leader_election.cc:288] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 3 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:15:58.425215 11173 raft_consensus.cc:2781] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:15:58.434916  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a3dcc2de594c8982477a0229f71191" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "9e699d9f984b4cf2b8b468322759b78c"
I20240503 02:15:58.435159  4326 raft_consensus.cc:3037] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:15:58.439343  4326 raft_consensus.cc:2454] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 3.
I20240503 02:15:58.444346 11173 raft_consensus.cc:686] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 3 LEADER]: Becoming Leader. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Running, Role: LEADER
I20240503 02:15:58.445108 11173 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:15:58.476603  4592 raft_consensus.cc:1203] T 8870feaf47a64a929dd2e584ece47218 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:15:58.498486  2136 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: term changed from 2 to 3, leader changed from 08055479ea104aa290d19955d3720850 (127.20.218.65) to 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69). New cstate: current_term: 3 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:15:58.812486  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:58.929697  4176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:58.929769  2279 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 3 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:15:58.930577 11173 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:58.944360  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:58.948555  4326 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:15:58.949611 11173 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:15:58.992058  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:59.007586  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.026167  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:59.045933  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:59.046556  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 79488.5us
I20240503 02:15:59.079936  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:15:59.085012  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 81962.5us
W20240503 02:15:59.085989  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 49238.8us
I20240503 02:15:59.089735  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:59.114269  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:59.127089  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:59.131047  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.131805  2391 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:59.132222  2390 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:59.178490  4176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:59.186800  4176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:59.230856  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.231292  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:59.232244  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:59.256498  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.257272  2390 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:59.269965  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.398207  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:59.402428  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.659121  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:59.677871  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.689677  2390 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:15:59.694595  4176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:59.740365  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:59.766080  4176 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:15:59.783754  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.796173  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:15:59.818061  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:15:59.845350  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:15:59.896206  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.921453  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.945756  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:15:59.999596  2390 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:16:00.037565  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:00.079918  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:00.205869  4196 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:00.217473  4326 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240503 02:16:00.219161 11287 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:16:00.251370  2412 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.264498 11411 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:00.267309  4327 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:00.277773  2138 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 8 to 10, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 3 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:00.289552  2410 consensus_queue.cc:237] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:00.302204  4460 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:16:00.303767 11285 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:00.320637 11285 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:00.340178  4460 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:00.350883  4196 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:16:00.352460 10374 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:00.355764  4327 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:16:00.357445 11285 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:00.396612  2135 catalog_manager.cc:5484] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:00.413552  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a3dcc2de594c8982477a0229f71191 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38192
I20240503 02:16:00.429127  2261 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f7c9c29384949fb95cb0c3b765f57b0 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:16:00.430035 11441 tablet_replica.cc:330] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:16:00.430385 11441 raft_consensus.cc:2229] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:00.430662 11441 raft_consensus.cc:2258] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:00.431736 11441 ts_tablet_manager.cc:1875] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:00.390915 11285 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.440423  4196 consensus_queue.cc:237] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:00.440352  4327 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.443557  4325 raft_consensus.cc:1261] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240503 02:16:00.444185 11287 consensus_queue.cc:1035] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:16:00.447062 11411 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:00.451268 11441 ts_tablet_manager.cc:1888] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240503 02:16:00.452361  4325 raft_consensus.cc:2932] T 62a3dcc2de594c8982477a0229f71191 P 9e699d9f984b4cf2b8b468322759b78c [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
W20240503 02:16:00.453573  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 81263.2us
I20240503 02:16:00.454545  4592 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:16:00.454660 11441 log.cc:1196] T 62a3dcc2de594c8982477a0229f71191 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/62a3dcc2de594c8982477a0229f71191
I20240503 02:16:00.455574 11441 tablet_replica.cc:330] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:16:00.455605  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 62a3dcc2de594c8982477a0229f71191 (table rebalance_test_table_6 [id=2d127e4ea62a4b498d1efd3a86c450c6]) successfully deleted
I20240503 02:16:00.455821 11441 raft_consensus.cc:2229] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:00.456048 11441 raft_consensus.cc:2258] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:00.457015 11441 ts_tablet_manager.cc:1875] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:00.456948  2412 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "771a4190142b4fd5a9479417122c0881"
dest_uuid: "6d54feb0bbc14c109783371d2939b234"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60302
I20240503 02:16:00.457216  2412 raft_consensus.cc:595] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:16:00.462708  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 62a3dcc2de594c8982477a0229f71191 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:00.468202  2135 catalog_manager.cc:5484] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 10 to 11, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 3 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:00.472210 11427 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20240503 02:16:00.483752  4132 consensus_peers.cc:485] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a3dcc2de594c8982477a0229f71191. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:00.487820 11441 ts_tablet_manager.cc:1888] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:16:00.489169 11441 log.cc:1196] T 9f7c9c29384949fb95cb0c3b765f57b0 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/9f7c9c29384949fb95cb0c3b765f57b0
I20240503 02:16:00.490219  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 9f7c9c29384949fb95cb0c3b765f57b0 (table rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]) successfully deleted
I20240503 02:16:00.492514  2139 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
W20240503 02:16:00.493095  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 101718us
I20240503 02:16:00.498760  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b822c691d824b2ba25e967535c9734e with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:36984
I20240503 02:16:00.500535  2412 consensus_queue.cc:237] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.504402  4196 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.505311  4460 raft_consensus.cc:1261] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:16:00.506352 11450 consensus_queue.cc:1035] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:00.509876  4592 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
W20240503 02:16:00.518488  2347 consensus_peers.cc:485] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f7c9c29384949fb95cb0c3b765f57b0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:00.520134 11427 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.525534 11457 tablet_replica.cc:330] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:16:00.525969 11457 raft_consensus.cc:2229] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:00.526325 11457 raft_consensus.cc:2258] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:00.529851 11457 ts_tablet_manager.cc:1875] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:00.540143  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9f7c9c29384949fb95cb0c3b765f57b0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:00.540822  4460 raft_consensus.cc:2932] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.541653  2135 catalog_manager.cc:5484] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 6 to 7, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.551252 11457 ts_tablet_manager.cc:1888] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:16:00.551419 11457 log.cc:1196] T 9b822c691d824b2ba25e967535c9734e P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b822c691d824b2ba25e967535c9734e
I20240503 02:16:00.552321  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 9b822c691d824b2ba25e967535c9734e (table rebalance_test_table_3 [id=448499f38b724268990226d70f70f241]) successfully deleted
I20240503 02:16:00.715881  2540 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:16:00.724301  2412 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:16:00.725067 11308 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:00.732519 11427 raft_consensus.cc:979] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: : Instructing follower 1d36807868fe4a82bca42c25831ce8a2 to start an election
I20240503 02:16:00.732689 11427 raft_consensus.cc:1067] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Signalling peer 1d36807868fe4a82bca42c25831ce8a2 to start an election
I20240503 02:16:00.733659  4196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "771a4190142b4fd5a9479417122c0881"
dest_uuid: "1d36807868fe4a82bca42c25831ce8a2"
 from {username='slave'} at 127.20.218.66:32795
I20240503 02:16:00.733915  4196 raft_consensus.cc:482] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:00.734036  4196 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:00.736994  4196 raft_consensus.cc:504] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.738700  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:16:00.738942  2410 raft_consensus.cc:3032] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:00.739023  2410 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:16:00.739238  4196 leader_election.cc:288] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:16:00.739329  2410 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.739681  2410 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:00.743724  2410 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 2.
I20240503 02:16:00.744360  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a4190142b4fd5a9479417122c0881" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:16:00.744451  4129 leader_election.cc:302] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:16:00.744647  2280 raft_consensus.cc:3037] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:00.744856 11411 raft_consensus.cc:2781] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:00.744980 11411 raft_consensus.cc:686] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Becoming Leader. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Running, Role: LEADER
I20240503 02:16:00.745306 11411 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.748610  2280 raft_consensus.cc:2454] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 2.
I20240503 02:16:00.790060  2135 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: term changed from 1 to 2, leader changed from 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) to 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69). New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.792538 11308 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:16:00.794754  2410 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:16:00.805109  2135 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:16:00.810817  4573 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec063c94fe941ebb1c866b5b8935ece with delete_type TABLET_DATA_TOMBSTONED (TS 7fd07044862a481aa387698017467c01 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34002
I20240503 02:16:00.814713  2540 consensus_queue.cc:237] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:00.817859  2410 raft_consensus.cc:1261] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:16:00.818327  2477 consensus_peers.cc:485] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Couldn't send request to peer 1d36807868fe4a82bca42c25831ce8a2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ec063c94fe941ebb1c866b5b8935ece. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:00.818531 10980 consensus_queue.cc:1035] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:16:00.823556 10980 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.828794 11518 tablet_replica.cc:330] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: stopping tablet replica
I20240503 02:16:00.829211  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1ec063c94fe941ebb1c866b5b8935ece with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:00.829283 11518 raft_consensus.cc:2229] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:00.829607 11518 raft_consensus.cc:2258] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:00.830632  2410 raft_consensus.cc:2932] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.839201  2135 catalog_manager.cc:5484] T 1ec063c94fe941ebb1c866b5b8935ece P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 8 to 9, VOTER 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:00.840894 11518 ts_tablet_manager.cc:1875] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:00.856175 11518 ts_tablet_manager.cc:1888] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240503 02:16:00.860711 11518 log.cc:1196] T 1ec063c94fe941ebb1c866b5b8935ece P 7fd07044862a481aa387698017467c01: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-7/wal/wals/1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:16:00.862255  2126 catalog_manager.cc:4830] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): tablet 1ec063c94fe941ebb1c866b5b8935ece (table rebalance_test_table_9 [id=263a2ae2aba54460aee371a910b0c410]) successfully deleted
I20240503 02:16:00.869693 11534 ts_tablet_manager.cc:902] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:16:00.888983  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 70742.9us
I20240503 02:16:00.997366 11534 tablet_copy_client.cc:308] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.69:39741
I20240503 02:16:01.052177 11580 ts_tablet_manager.cc:902] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:16:01.056759  4216 tablet_copy_service.cc:140] P 1d36807868fe4a82bca42c25831ce8a2: Received BeginTabletCopySession request for tablet 62a3dcc2de594c8982477a0229f71191 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:41757)
I20240503 02:16:01.057051  4216 tablet_copy_service.cc:161] P 1d36807868fe4a82bca42c25831ce8a2: Beginning new tablet copy session on tablet 62a3dcc2de594c8982477a0229f71191 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:41757: session id = 408af0ab5520465e942cc714bfc6c73b-62a3dcc2de594c8982477a0229f71191
I20240503 02:16:01.061908  4216 tablet_copy_source_session.cc:205] T 62a3dcc2de594c8982477a0229f71191 P 1d36807868fe4a82bca42c25831ce8a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:01.065215 11534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a3dcc2de594c8982477a0229f71191. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:01.101927 11534 tablet_copy_client.cc:783] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:01.102722 11534 tablet_copy_client.cc:647] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:01.105919 11534 tablet_copy_client.cc:523] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:01.116398 11534 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:16:01.185537 11580 tablet_copy_client.cc:308] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:16:01.186605  2410 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:16:01.189405  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 9f7c9c29384949fb95cb0c3b765f57b0 from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:49243)
I20240503 02:16:01.189582  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 9f7c9c29384949fb95cb0c3b765f57b0 from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:49243: session id = 1d36807868fe4a82bca42c25831ce8a2-9f7c9c29384949fb95cb0c3b765f57b0
I20240503 02:16:01.192296  2432 tablet_copy_source_session.cc:205] T 9f7c9c29384949fb95cb0c3b765f57b0 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:01.192281 11411 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:01.195056  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:01.207234  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:01.260890  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.262975  2390 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:16:01.269842 11636 ts_tablet_manager.cc:902] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:16:01.298614 11580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f7c9c29384949fb95cb0c3b765f57b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:01.315097  2280 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:16:01.315924 11287 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:01.336759  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:01.356756 11534 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:01.357267 11534 tablet_bootstrap.cc:492] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:16:01.391487 11534 ts_tablet_manager.cc:1367] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.275s	user 0.073s	sys 0.023s
I20240503 02:16:01.392524 11534 raft_consensus.cc:348] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:01.392807 11534 raft_consensus.cc:729] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:16:01.402940 11534 consensus_queue.cc:260] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:01.404673  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:01.406302  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.405891 11534 ts_tablet_manager.cc:1398] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20240503 02:16:01.409759  4216 tablet_copy_service.cc:342] P 1d36807868fe4a82bca42c25831ce8a2: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-62a3dcc2de594c8982477a0229f71191 received from {username='slave'} at 127.20.218.71:41757
I20240503 02:16:01.409948  4216 tablet_copy_service.cc:433] P 1d36807868fe4a82bca42c25831ce8a2: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-62a3dcc2de594c8982477a0229f71191 on tablet 62a3dcc2de594c8982477a0229f71191 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:16:01.417100  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:01.438925  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:01.443792  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.479787  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.498246  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.510223  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.517225  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:01.525008  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:01.554214 11580 tablet_copy_client.cc:783] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:01.554569 11580 tablet_copy_client.cc:647] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:01.558105 11580 tablet_copy_client.cc:523] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:16:01.560501  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 72872.7us
I20240503 02:16:01.565018 11636 tablet_copy_client.cc:308] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:16:01.565753  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 1d36807868fe4a82bca42c25831ce8a2 ({username='slave'} at 127.20.218.69:57071)
I20240503 02:16:01.565901  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 1ec063c94fe941ebb1c866b5b8935ece from peer 1d36807868fe4a82bca42c25831ce8a2 at {username='slave'} at 127.20.218.69:57071: session id = 1d36807868fe4a82bca42c25831ce8a2-1ec063c94fe941ebb1c866b5b8935ece
I20240503 02:16:01.568892  2562 tablet_copy_source_session.cc:205] T 1ec063c94fe941ebb1c866b5b8935ece P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:01.569998  2390 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:16:01.572870 11636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec063c94fe941ebb1c866b5b8935ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:01.586570 11580 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:16:01.622792 11636 tablet_copy_client.cc:783] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:01.623230 11636 tablet_copy_client.cc:647] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:01.652451 11636 tablet_copy_client.cc:523] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:01.677947 11636 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap starting.
I20240503 02:16:01.801491  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:01.824859  2520 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:01.826097  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.849457  2390 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:16:01.880681  4306 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:01.885157  2651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:01.914300  4460 raft_consensus.cc:1203] T 62a3dcc2de594c8982477a0229f71191 P 408af0ab5520465e942cc714bfc6c73b [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11]   Dedup: 3.11->[]
I20240503 02:16:01.927984  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:01.935191  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:01.968449  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 57506.9us
I20240503 02:16:02.086427  2540 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4"
dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47540
I20240503 02:16:02.086805  2540 raft_consensus.cc:595] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:16:02.118616 11670 raft_consensus.cc:979] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: : Instructing follower 9e699d9f984b4cf2b8b468322759b78c to start an election
I20240503 02:16:02.121273 11521 raft_consensus.cc:1067] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Signalling peer 9e699d9f984b4cf2b8b468322759b78c to start an election
I20240503 02:16:02.122364  4327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4"
dest_uuid: "9e699d9f984b4cf2b8b468322759b78c"
 from {username='slave'} at 127.20.218.67:56343
I20240503 02:16:02.122750  4327 raft_consensus.cc:482] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:02.122901  4327 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.126578  4327 raft_consensus.cc:504] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.129659  2540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "9e699d9f984b4cf2b8b468322759b78c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
I20240503 02:16:02.129926  2540 raft_consensus.cc:3032] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:02.129999  2540 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:16:02.130352  2540 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.130828  2540 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.135284  2540 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e699d9f984b4cf2b8b468322759b78c in term 2.
I20240503 02:16:02.136412  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "9e699d9f984b4cf2b8b468322759b78c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:16:02.136708  2410 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.140105  2410 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e699d9f984b4cf2b8b468322759b78c in term 2.
I20240503 02:16:02.140851  4260 leader_election.cc:302] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234, 9e699d9f984b4cf2b8b468322759b78c; no voters: 
I20240503 02:16:02.152387  4327 leader_election.cc:288] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 election: Requested vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 7fd07044862a481aa387698017467c01 (127.20.218.72:37033)
I20240503 02:16:02.165809 11580 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:02.166299 11580 tablet_bootstrap.cc:492] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:16:02.166574 11580 ts_tablet_manager.cc:1367] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.580s	user 0.066s	sys 0.005s
I20240503 02:16:02.167356 11580 raft_consensus.cc:348] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:02.167637 11580 raft_consensus.cc:729] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:16:02.167889 11580 consensus_queue.cc:260] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:02.168596 11580 ts_tablet_manager.cc:1398] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:16:02.169647 11757 raft_consensus.cc:2781] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:02.171579  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-9f7c9c29384949fb95cb0c3b765f57b0 received from {username='slave'} at 127.20.218.69:49243
I20240503 02:16:02.171706  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-9f7c9c29384949fb95cb0c3b765f57b0 on tablet 9f7c9c29384949fb95cb0c3b765f57b0 with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:16:02.212705  4591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "592cf4d5b35e49bfa4173343d64367f4" candidate_uuid: "9e699d9f984b4cf2b8b468322759b78c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7fd07044862a481aa387698017467c01"
I20240503 02:16:02.212958  4591 raft_consensus.cc:3037] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.216197  4591 raft_consensus.cc:2454] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e699d9f984b4cf2b8b468322759b78c in term 2.
I20240503 02:16:02.247115 11757 raft_consensus.cc:686] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 LEADER]: Becoming Leader. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Running, Role: LEADER
I20240503 02:16:02.247743 11757 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.248171 11636 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:02.248665 11636 tablet_bootstrap.cc:492] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Bootstrap complete.
I20240503 02:16:02.248924 11636 ts_tablet_manager.cc:1367] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Time spent bootstrapping tablet: real 0.571s	user 0.064s	sys 0.001s
I20240503 02:16:02.257709 11636 raft_consensus.cc:348] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:02.257952 11636 raft_consensus.cc:729] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Initialized, Role: FOLLOWER
I20240503 02:16:02.258184 11636 consensus_queue.cc:260] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:02.260047 11636 ts_tablet_manager.cc:1398] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.000s
I20240503 02:16:02.260057  2135 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: term changed from 1 to 2, leader changed from 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) to 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70). New cstate: current_term: 2 leader_uuid: "9e699d9f984b4cf2b8b468322759b78c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:02.264387  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 1d36807868fe4a82bca42c25831ce8a2-1ec063c94fe941ebb1c866b5b8935ece received from {username='slave'} at 127.20.218.69:57071
I20240503 02:16:02.264535  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 1d36807868fe4a82bca42c25831ce8a2-1ec063c94fe941ebb1c866b5b8935ece on tablet 1ec063c94fe941ebb1c866b5b8935ece with peer 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:16:02.337751  4196 raft_consensus.cc:1203] T 9f7c9c29384949fb95cb0c3b765f57b0 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:16:02.360522  2410 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0137e646f1bd452b950d870107afa060"
dest_uuid: "6d54feb0bbc14c109783371d2939b234"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60370
I20240503 02:16:02.360916  2410 raft_consensus.cc:595] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:16:02.374264  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:02.386168  2280 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:02.386793 11411 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:02.401052 11808 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:02.402222  2280 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:02.412079  2135 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:02.419534  2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a4190142b4fd5a9479417122c0881 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52816
I20240503 02:16:02.422333  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
W20240503 02:16:02.433079  4131 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Couldn't send request to peer e9fdaf326bb64edabc43b366df870f83. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:02.435550  2280 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:02.436838 11817 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 02:16:02.440416 11817 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:02.440873  2280 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:02.447415 11823 tablet_replica.cc:330] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:16:02.447536  2135 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 7 to 8, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:02.448604  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 771a4190142b4fd5a9479417122c0881 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:02.452600 11823 raft_consensus.cc:2229] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:02.454135 11823 raft_consensus.cc:2258] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:02.455197 11823 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:02.471468 11823 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:16:02.471946 11823 log.cc:1196] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/771a4190142b4fd5a9479417122c0881
I20240503 02:16:02.478072  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 771a4190142b4fd5a9479417122c0881 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
I20240503 02:16:02.501566 11285 raft_consensus.cc:979] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: : Instructing follower e9fdaf326bb64edabc43b366df870f83 to start an election
I20240503 02:16:02.501725 11285 raft_consensus.cc:1067] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Signalling peer e9fdaf326bb64edabc43b366df870f83 to start an election
I20240503 02:16:02.502641  2669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0137e646f1bd452b950d870107afa060"
dest_uuid: "e9fdaf326bb64edabc43b366df870f83"
 from {username='slave'} at 127.20.218.66:53211
I20240503 02:16:02.502961  2669 raft_consensus.cc:482] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:02.503079  2669 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.509256  2669 raft_consensus.cc:504] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.511544  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:16:02.511822  2410 raft_consensus.cc:3032] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:02.511907  2410 raft_consensus.cc:729] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:16:02.512246  2410 consensus_queue.cc:260] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.512763  2410 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.516615  2410 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 2.
I20240503 02:16:02.519580  4460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "408af0ab5520465e942cc714bfc6c73b"
I20240503 02:16:02.519904  4460 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.530648  4460 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 2.
I20240503 02:16:02.533074  2609 leader_election.cc:302] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b, 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83; no voters: 
I20240503 02:16:02.533564 11667 raft_consensus.cc:2781] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:02.533711 11667 raft_consensus.cc:686] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 2 LEADER]: Becoming Leader. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Running, Role: LEADER
I20240503 02:16:02.534103 11667 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:02.539407  2135 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: term changed from 1 to 2, leader changed from 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) to e9fdaf326bb64edabc43b366df870f83 (127.20.218.68). New cstate: current_term: 2 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:02.548159  2669 leader_election.cc:288] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:16:02.548221  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "9e699d9f984b4cf2b8b468322759b78c"
I20240503 02:16:02.548425  4326 raft_consensus.cc:3037] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:02.551229  4326 raft_consensus.cc:2454] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9fdaf326bb64edabc43b366df870f83 in term 2.
I20240503 02:16:02.734099  2540 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:02.734885 11757 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:02.750229  2410 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:02.751312 11757 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:02.802595  4591 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:02.805199 11757 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:02.831594  4196 raft_consensus.cc:1203] T 1ec063c94fe941ebb1c866b5b8935ece P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:16:02.986243 11941 ts_tablet_manager.cc:902] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Initiating tablet copy from peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:16:02.992319  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:03.005540  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.018913  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:03.029888  2410 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:03.031286 11667 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:03.049285 11941 tablet_copy_client.cc:308] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.69:39741
I20240503 02:16:03.050220  4216 tablet_copy_service.cc:140] P 1d36807868fe4a82bca42c25831ce8a2: Received BeginTabletCopySession request for tablet 771a4190142b4fd5a9479417122c0881 from peer e9fdaf326bb64edabc43b366df870f83 ({username='slave'} at 127.20.218.68:39891)
I20240503 02:16:03.050386  4216 tablet_copy_service.cc:161] P 1d36807868fe4a82bca42c25831ce8a2: Beginning new tablet copy session on tablet 771a4190142b4fd5a9479417122c0881 from peer e9fdaf326bb64edabc43b366df870f83 at {username='slave'} at 127.20.218.68:39891: session id = e9fdaf326bb64edabc43b366df870f83-771a4190142b4fd5a9479417122c0881
I20240503 02:16:03.050908  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.052923  4326 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:03.053241  4216 tablet_copy_source_session.cc:205] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:03.053560 11667 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:03.055632 11941 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:16:03.061303  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.087713  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:03.106204 11941 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:16:03.106570 11941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:03.116766  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:03.133194  4459 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:03.141064 11667 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:03.143196 11941 tablet_copy_client.cc:783] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:03.143548 11941 tablet_copy_client.cc:647] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:03.152429 11941 tablet_copy_client.cc:523] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:03.156859 11941 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:16:03.202884  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.315274  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.364392 11941 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:03.364885 11941 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Bootstrap complete.
I20240503 02:16:03.365141 11941 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.208s	user 0.055s	sys 0.010s
I20240503 02:16:03.366158 11941 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:16:03.366364 11941 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:16:03.366628 11941 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:16:03.373692  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:03.367591 11941 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:16:03.393194  4216 tablet_copy_service.cc:342] P 1d36807868fe4a82bca42c25831ce8a2: Request end of tablet copy session e9fdaf326bb64edabc43b366df870f83-771a4190142b4fd5a9479417122c0881 received from {username='slave'} at 127.20.218.68:39891
I20240503 02:16:03.393368  4216 tablet_copy_service.cc:433] P 1d36807868fe4a82bca42c25831ce8a2: ending tablet copy session e9fdaf326bb64edabc43b366df870f83-771a4190142b4fd5a9479417122c0881 on tablet 771a4190142b4fd5a9479417122c0881 with peer e9fdaf326bb64edabc43b366df870f83
I20240503 02:16:03.393563  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.408411  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:03.414708  2669 raft_consensus.cc:1203] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:16:03.461058  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:03.488130  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:03.504565  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.539458  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.611297  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:03.646448  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:03.647522  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:03.675446  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.686069  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.714254  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.746114  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.776646  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:03.787508  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:03.801824  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:03.805766  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:03.822599  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:03.869966  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.894662  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:03.902611  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:03.938212  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:04.147249  4326 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:04.150838  4591 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:04.152477 11908 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:04.159159  2410 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:04.160076 11908 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:04.175688 12123 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.184875  2412 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.193293  2138 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 6 to 8, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "9e699d9f984b4cf2b8b468322759b78c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.200325  4592 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 7fd07044862a481aa387698017467c01 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.264048  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet 592cf4d5b35e49bfa4173343d64367f4 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47402
I20240503 02:16:04.300123 12155 tablet_replica.cc:330] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:16:04.300345 12155 raft_consensus.cc:2229] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:04.300527 12155 raft_consensus.cc:2258] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:04.301223 12155 ts_tablet_manager.cc:1875] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:04.325212 12155 ts_tablet_manager.cc:1888] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:16:04.327925 12155 log.cc:1196] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/592cf4d5b35e49bfa4173343d64367f4
I20240503 02:16:04.329224  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet 592cf4d5b35e49bfa4173343d64367f4 (table rebalance_test_table_7 [id=fc0f9aa11af74e438eac8b71019ea20b]) successfully deleted
I20240503 02:16:04.334198  2540 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:04.349495  4326 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:16:04.351034 11915 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:04.378794 12086 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.382792  4326 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.391732  2139 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 5 to 6, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.405164  2540 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
W20240503 02:16:04.407972  2477 consensus_peers.cc:485] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51dbf0fd54b14c2fa21d3e62104c9b87. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:04.408607  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38192
I20240503 02:16:04.410108  4326 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:16:04.410957 11915 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:04.413980 11915 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:16:04.418239  4326 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:16:04.434051  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:04.436937  2131 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } } }
I20240503 02:16:04.454602 12191 tablet_replica.cc:330] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:16:04.455093 12191 raft_consensus.cc:2229] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:04.455415 12191 raft_consensus.cc:2258] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:04.456670 12191 ts_tablet_manager.cc:1875] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:04.486689 12191 ts_tablet_manager.cc:1888] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:16:04.496137 12191 log.cc:1196] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/51dbf0fd54b14c2fa21d3e62104c9b87
I20240503 02:16:04.497964  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 51dbf0fd54b14c2fa21d3e62104c9b87 (table rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]) successfully deleted
I20240503 02:16:04.515173  2669 consensus_queue.cc:237] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:04.553119  4327 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:04.554209 11959 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:04.569198  4459 raft_consensus.cc:1261] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:04.570735 11959 consensus_queue.cc:1035] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:04.594427 12131 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P e9fdaf326bb64edabc43b366df870f83 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.595363  4460 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.601720  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:04.630221  2131 catalog_manager.cc:5484] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 6 to 8, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.636907  2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 0137e646f1bd452b950d870107afa060 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:52816
I20240503 02:16:04.646875  4326 raft_consensus.cc:2932] T 0137e646f1bd452b950d870107afa060 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:04.663290  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:04.666281 12236 tablet_replica.cc:330] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:16:04.673470  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:04.678718 12236 raft_consensus.cc:2229] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:04.679780  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:04.689483 12236 raft_consensus.cc:2258] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:04.700316  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:04.725917  4573 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:04.735329 12236 ts_tablet_manager.cc:1875] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:04.753008 12236 ts_tablet_manager.cc:1888] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:16:04.753166 12236 log.cc:1196] T 0137e646f1bd452b950d870107afa060 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/0137e646f1bd452b950d870107afa060
I20240503 02:16:04.794488  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:04.795693  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 0137e646f1bd452b950d870107afa060 (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
W20240503 02:16:04.909090  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 145491us
I20240503 02:16:04.955933  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:05.039314 12283 ts_tablet_manager.cc:902] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 7fd07044862a481aa387698017467c01: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
F20240503 02:16:05.039898 12283 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.queue_time_us, tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.queue_time_us) (tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.run_cpu_time_us, tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.run_cpu_time_us) (tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.run_wall_time_us, tablet-download-pool-02e0b954008c4f30af414da6cb5d59c2.run_wall_time_us) (tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.queue_time_us, tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.queue_time_us) (tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.run_cpu_time_us, tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.run_cpu_time_us) (tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.run_wall_time_us, tablet-download-pool-0e61eb7d9e6041b6b99d5d5328943140.run_wall_time_us) (tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.queue_time_us, tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.queue_time_us) (tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.run_cpu_time_us, tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.run_cpu_time_us) (tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.run_wall_time_us, tablet-download-pool-190ebb1335d64b279a41fa38feff22b4.run_wall_time_us) (tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.queue_time_us, tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.queue_time_us) (tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.run_cpu_time_us, tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.run_cpu_time_us) (tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.run_wall_time_us, tablet-download-pool-1ec063c94fe941ebb1c866b5b8935ece.run_wall_time_us) (tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.queue_time_us, tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.queue_time_us) (tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.run_cpu_time_us, tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.run_cpu_time_us) (tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.run_wall_time_us, tablet-download-pool-1eeb69e5eb3c459ca749aa37672ec057.run_wall_time_us) (tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.queue_time_us, tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.queue_time_us) (tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.run_cpu_time_us, tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.run_cpu_time_us) (tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.run_wall_time_us, tablet-download-pool-2de1d29dbea945eba58f8eadb019d766.run_wall_time_us) (tablet-download-pool-51dbf0fd54b14c2fa21d3e62104c9b87.queue_time_us, tablet-download-pool-51dbf0fd54b14c2fa21d3e62104c9b87.queue_time_us) (tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.queue_time_us, tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.queue_time_us) (tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.run_cpu_time_us, tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.run_cpu_time_us) (tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.run_wall_time_us, tablet-download-pool-592cf4d5b35e49bfa4173343d64367f4.run_wall_time_us) (tablet-download-pool-8870feaf47a64a929dd2e584ece47218.queue_time_us, tablet-download-pool-8870feaf47a64a929dd2e584ece47218.queue_time_us) (tablet-download-pool-8870feaf47a64a929dd2e584ece47218.run_cpu_time_us, tablet-download-pool-8870feaf47a64a929dd2e584ece47218.run_cpu_time_us) (tablet-download-pool-8870feaf47a64a929dd2e584ece47218.run_wall_time_us, tablet-download-pool-8870feaf47a64a929dd2e584ece47218.run_wall_time_us) (tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.queue_time_us, tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.queue_time_us) (tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.run_cpu_time_us, tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.run_cpu_time_us) (tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.run_wall_time_us, tablet-download-pool-913d0aa014ad4bcaac905b5fdaa296d2.run_wall_time_us) (tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.queue_time_us, tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.queue_time_us) (tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.run_cpu_time_us, tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.run_cpu_time_us) (tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.run_wall_time_us, tablet-download-pool-9253de3f798f4a208d566765ed2ba54d.run_wall_time_us) (tablet-download-pool-9b822c691d824b2ba25e967535c9734e.queue_time_us, tablet-download-pool-9b822c691d824b2ba25e967535c9734e.queue_time_us) (tablet-download-pool-9b822c691d824b2ba25e967535c9734e.run_cpu_time_us, tablet-download-pool-9b822c691d824b2ba25e967535c9734e.run_cpu_time_us) (tablet-download-pool-9b822c691d824b2ba25e967535c9734e.run_wall_time_us, tablet-download-pool-9b822c691d824b2ba25e967535c9734e.run_wall_time_us) (tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.queue_time_us, tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.queue_time_us) (tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.run_cpu_time_us, tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.run_cpu_time_us) (tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.run_wall_time_us, tablet-download-pool-a1949a5c7cc741b19da021eccb7d0dd1.run_wall_time_us) (tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.queue_time_us, tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.queue_time_us) (tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.run_cpu_time_us, tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.run_cpu_time_us) (tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.run_wall_time_us, tablet-download-pool-a7ab8b0c8db54565a6066b9a7d1569d8.run_wall_time_us) (tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.queue_time_us, tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.queue_time_us) (tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.run_cpu_time_us, tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.run_cpu_time_us) (tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.run_wall_time_us, tablet-download-pool-d2351807a2ed47419f8d674fbeb9d0ec.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7fed09dfcdcd  google::LogMessage::Fail() at ??:0
    @     0x7fed09e00b93  google::LogMessage::SendToLog() at ??:0
    @     0x7fed09dfc7cc  google::LogMessage::Flush() at ??:0
    @     0x7fed09dfdf59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fed0cb3d742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7fed0cb09f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7fed0cb03cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fed1bf578de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7fed1bf6fdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7fed1c09a480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7fed1c0d2a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fed1c0b5981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7fed1c0b570e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fed0cb0db75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7fed0cb1119c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7fed0cb11019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fed0caec01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fed0fe716db  start_thread at ??:0
    @     0x7fed08caf71f  clone at ??:0
I20240503 02:16:05.455523  2281 consensus_queue.cc:237] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:05.499588  4196 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:16:05.500413 11578 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:05.534008  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:05.578934 12393 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:05.624855  4196 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:05.632464  2131 catalog_manager.cc:5484] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 5 to 6, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:05.679893  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:05.866137  2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 734027e195414b15811ac9d32b132db8 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52816
I20240503 02:16:05.868264  2281 consensus_queue.cc:237] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
W20240503 02:16:05.872887  2217 consensus_peers.cc:485] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 734027e195414b15811ac9d32b132db8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:05.874017  4196 raft_consensus.cc:1261] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:16:05.874840 12395 consensus_queue.cc:1035] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:05.878103 12393 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:05.885419  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:05.889443  2127 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 734027e195414b15811ac9d32b132db8 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:05.890614  4196 raft_consensus.cc:2932] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:05.894886  2139 catalog_manager.cc:5484] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 6 to 7, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:05.914261  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:05.916533 12445 tablet_replica.cc:330] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:16:05.916851 12445 raft_consensus.cc:2229] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:05.932565  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:05.934837 12445 raft_consensus.cc:2258] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:05.947270  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:05.951848  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:16:05.957522  4262 connection.cc:535] client connection to 127.20.218.72:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.957789  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:05.960225 12445 ts_tablet_manager.cc:1875] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:16:05.964195  2216 connection.cc:535] client connection to 127.20.218.72:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.964442  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.964581  2608 connection.cc:535] client connection to 127.20.218.72:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.964886  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:16:05.966651  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:05.979308  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:05.989462  2477 connection.cc:535] client connection to 127.20.218.72:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.989814  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.963763  2347 connection.cc:535] client connection to 127.20.218.72:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:05.993641  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:16:06.015750  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 51368.6us
I20240503 02:16:06.021734  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:06.032040 12445 ts_tablet_manager.cc:1888] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:16:06.032202 12445 log.cc:1196] T 734027e195414b15811ac9d32b132db8 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/734027e195414b15811ac9d32b132db8
I20240503 02:16:06.033160  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 734027e195414b15811ac9d32b132db8 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:16:06.040366  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:06.042367  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.045302  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.045485  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.045574  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.045663  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:06.053558  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:06.056154  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.056314  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.056382  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:06.063234  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:06.089705  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 99966.7us
I20240503 02:16:06.094230  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:06.103237  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:06.103436  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.103567  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:06.103662  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:06.125562  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:06.126252  4306 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:06.126494  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:06.173589  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:06.181311  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:06.207458  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:06.269410  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:06.301398  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:06.366976 12503 ts_tablet_manager.cc:902] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:16:06.370040  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:06.451265 12503 tablet_copy_client.cc:308] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:16:06.452072  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 734027e195414b15811ac9d32b132db8 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:56489)
I20240503 02:16:06.452253  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 734027e195414b15811ac9d32b132db8 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:56489: session id = 408af0ab5520465e942cc714bfc6c73b-734027e195414b15811ac9d32b132db8
I20240503 02:16:06.455307  2301 tablet_copy_source_session.cc:205] T 734027e195414b15811ac9d32b132db8 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:06.457877 12503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 734027e195414b15811ac9d32b132db8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:06.468304  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:06.494282  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:06.501089  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:06.519950 12503 tablet_copy_client.cc:783] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:06.520303 12503 tablet_copy_client.cc:647] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:06.522413 12503 tablet_copy_client.cc:523] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:06.530949 12503 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:16:06.560711  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:06.584944  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:06.588860  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:06.597924  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:06.829676  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:06.900835 12503 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:06.901860 12503 tablet_bootstrap.cc:492] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:16:06.904007 12503 ts_tablet_manager.cc:1367] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.373s	user 0.053s	sys 0.007s
I20240503 02:16:06.905160 12503 raft_consensus.cc:348] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:06.905591 12503 raft_consensus.cc:729] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:16:06.905925 12503 consensus_queue.cc:260] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:06.961633 12503 ts_tablet_manager.cc:1398] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.057s	user 0.035s	sys 0.020s
I20240503 02:16:06.962656  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-734027e195414b15811ac9d32b132db8 received from {username='slave'} at 127.20.218.71:56489
I20240503 02:16:06.962791  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-734027e195414b15811ac9d32b132db8 on tablet 734027e195414b15811ac9d32b132db8 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:16:07.076915  2412 consensus_queue.cc:237] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:07.105228  4326 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:16:07.106070 12489 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:07.127054  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:07.128453 12489 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.129761  4327 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.137907  2131 catalog_manager.cc:5484] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 5 to 6, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.143281  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:07.169438  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:07.210191  2520 tablet_service.cc:1514] Processing DeleteTablet for tablet eadcb49d12664e6fb5a5ad8dbe47f342 with delete_type TABLET_DATA_TOMBSTONED (TS 08bef4c144bc4e4d8cb5836aee8da852 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47402
I20240503 02:16:07.216701  2412 consensus_queue.cc:237] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
W20240503 02:16:07.219276  2348 consensus_peers.cc:485] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eadcb49d12664e6fb5a5ad8dbe47f342. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:07.219942  4326 raft_consensus.cc:1261] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:16:07.220621 12489 consensus_queue.cc:1035] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:07.224454 12489 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:07.228688 12645 tablet_replica.cc:330] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: stopping tablet replica
I20240503 02:16:07.229076 12645 raft_consensus.cc:2229] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:07.228847  4326 raft_consensus.cc:2932] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:07.230578 12645 raft_consensus.cc:2258] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:07.232105 12645 ts_tablet_manager.cc:1875] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:07.236814  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet eadcb49d12664e6fb5a5ad8dbe47f342 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:07.237565  2131 catalog_manager.cc:5484] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 6 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:07.245841  4460 raft_consensus.cc:1203] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:16:07.250442  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:07.267802  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:07.278009  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:07.302866  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:07.307140 12645 ts_tablet_manager.cc:1888] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:16:07.308087 12645 log.cc:1196] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/eadcb49d12664e6fb5a5ad8dbe47f342
I20240503 02:16:07.309873  2125 catalog_manager.cc:4830] TS 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): tablet eadcb49d12664e6fb5a5ad8dbe47f342 (table rebalance_test_table_8 [id=92e4b9e7a87649ed8114a3f011cc6b56]) successfully deleted
W20240503 02:16:07.353506  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:07.504566  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:07.549531  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:07.555912  2281 consensus_queue.cc:237] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:16:07.564119  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:07.567845  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:07.571413  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:07.591392  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:07.592355  4326 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:16:07.593094 12393 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:16:07.615039 12393 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.618247  4327 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.628147  2133 catalog_manager.cc:5484] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 5 to 6, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:16:07.632795  2651 tablet_service.cc:1514] Processing DeleteTablet for tablet 586ebf467f0f427fa62dee6223a39848 with delete_type TABLET_DATA_TOMBSTONED (TS e9fdaf326bb64edabc43b366df870f83 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36984
I20240503 02:16:07.634783  2281 consensus_queue.cc:237] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
W20240503 02:16:07.637355  2214 consensus_peers.cc:485] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 586ebf467f0f427fa62dee6223a39848. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:07.639081  4326 raft_consensus.cc:1261] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:16:07.639720 12393 consensus_queue.cc:1035] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:07.642849 12621 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:16:07.650625  4327 raft_consensus.cc:2932] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:16:07.663933  2127 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 586ebf467f0f427fa62dee6223a39848 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:07.665618  2133 catalog_manager.cc:5484] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:16:07.671877  2652 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:07.728188 12706 tablet_replica.cc:330] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: stopping tablet replica
I20240503 02:16:07.728474 12706 raft_consensus.cc:2229] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:07.728699 12706 raft_consensus.cc:2258] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:07.729521 12706 ts_tablet_manager.cc:1875] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:07.773362 12706 ts_tablet_manager.cc:1888] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:16:07.785135 12720 ts_tablet_manager.cc:902] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:16:07.792263 12706 log.cc:1196] T 586ebf467f0f427fa62dee6223a39848 P e9fdaf326bb64edabc43b366df870f83: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/wals/586ebf467f0f427fa62dee6223a39848
I20240503 02:16:07.800889  2126 catalog_manager.cc:4830] TS e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): tablet 586ebf467f0f427fa62dee6223a39848 (table rebalance_test_table_5 [id=7ef4fa73f50841faa48a3fd6ec9d64a6]) successfully deleted
W20240503 02:16:07.849187  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:07.891212 12720 tablet_copy_client.cc:308] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:16:07.892289  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet eadcb49d12664e6fb5a5ad8dbe47f342 from peer 08055479ea104aa290d19955d3720850 ({username='slave'} at 127.20.218.65:33571)
I20240503 02:16:07.892446  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet eadcb49d12664e6fb5a5ad8dbe47f342 from peer 08055479ea104aa290d19955d3720850 at {username='slave'} at 127.20.218.65:33571: session id = 08055479ea104aa290d19955d3720850-eadcb49d12664e6fb5a5ad8dbe47f342
I20240503 02:16:07.895828  2432 tablet_copy_source_session.cc:205] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:07.898169 12720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eadcb49d12664e6fb5a5ad8dbe47f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:07.971690 12720 tablet_copy_client.cc:783] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:07.972136 12720 tablet_copy_client.cc:647] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:07.977041 12720 tablet_copy_client.cc:523] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:08.019898 12720 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
W20240503 02:16:08.079869  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:16:08.093763  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.113461  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:16:08.131700  2668 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:16:08.132486 11817 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:16:08.138087  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:16:08.142695  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.142953 12720 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:08.143514 12720 tablet_bootstrap.cc:492] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Bootstrap complete.
I20240503 02:16:08.144001 12720 ts_tablet_manager.cc:1367] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.124s	user 0.071s	sys 0.002s
I20240503 02:16:08.145318 12720 raft_consensus.cc:348] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:16:08.145617 12720 raft_consensus.cc:729] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:16:08.145962 12720 consensus_queue.cc:260] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:16:08.146732 12720 ts_tablet_manager.cc:1398] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.003s
I20240503 02:16:08.147599  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 08055479ea104aa290d19955d3720850-eadcb49d12664e6fb5a5ad8dbe47f342 received from {username='slave'} at 127.20.218.65:33571
I20240503 02:16:08.147742  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 08055479ea104aa290d19955d3720850-eadcb49d12664e6fb5a5ad8dbe47f342 on tablet eadcb49d12664e6fb5a5ad8dbe47f342 with peer 08055479ea104aa290d19955d3720850
I20240503 02:16:08.154711 12796 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:08.155575  2669 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:08.162165  2133 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 8 to 9, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:08.169502  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a4190142b4fd5a9479417122c0881 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38192
I20240503 02:16:08.174546  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:16:08.177565  2281 raft_consensus.cc:1203] T eadcb49d12664e6fb5a5ad8dbe47f342 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 02:16:08.185247  4129 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:16:08.198441  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.208815  2668 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:16:08.215034 11817 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:16:08.222330 12803 ts_tablet_manager.cc:902] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:16:08.229761 12798 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:16:08.230322  2669 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:16:08.233649  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 771a4190142b4fd5a9479417122c0881 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:08.238065  2133 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 9 to 10, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
W20240503 02:16:08.242492  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:16:08.249769  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:16:08.263057  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.264261 12801 tablet_replica.cc:330] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:16:08.264529 12801 raft_consensus.cc:2229] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:08.264964 12801 raft_consensus.cc:2258] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:08.266170 12801 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:16:08.305150  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.320676 12801 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
W20240503 02:16:08.322321  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.324626 12801 log.cc:1196] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/771a4190142b4fd5a9479417122c0881
I20240503 02:16:08.326951  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 771a4190142b4fd5a9479417122c0881 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
I20240503 02:16:08.331625 12803 tablet_copy_client.cc:308] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
W20240503 02:16:08.333525  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:08.336115  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 586ebf467f0f427fa62dee6223a39848 from peer 6d54feb0bbc14c109783371d2939b234 ({username='slave'} at 127.20.218.66:33413)
I20240503 02:16:08.336371  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 586ebf467f0f427fa62dee6223a39848 from peer 6d54feb0bbc14c109783371d2939b234 at {username='slave'} at 127.20.218.66:33413: session id = 6d54feb0bbc14c109783371d2939b234-586ebf467f0f427fa62dee6223a39848
I20240503 02:16:08.340587  2301 tablet_copy_source_session.cc:205] T 586ebf467f0f427fa62dee6223a39848 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:08.343603 12803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 586ebf467f0f427fa62dee6223a39848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:08.382154 12803 tablet_copy_client.cc:783] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:08.384776 12803 tablet_copy_client.cc:647] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:08.394769 12803 tablet_copy_client.cc:523] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:16:08.398128  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.408392 12803 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
W20240503 02:16:08.538213  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:16:08.589398  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:08.659515  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:08.678419  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:08.687942  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:08.693456 12913 ts_tablet_manager.cc:902] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Initiating tablet copy from peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:16:08.716286  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:08.723517  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:08.751299  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:08.756762  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:08.770223  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:08.773572  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 51490.9us
W20240503 02:16:08.819144 12803 log.cc:924] Time spent T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Append to log took a long time: real 0.088s	user 0.002s	sys 0.000s
I20240503 02:16:08.819679 12913 tablet_copy_client.cc:308] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.69:39741
I20240503 02:16:08.821116  4216 tablet_copy_service.cc:140] P 1d36807868fe4a82bca42c25831ce8a2: Received BeginTabletCopySession request for tablet 771a4190142b4fd5a9479417122c0881 from peer 6d54feb0bbc14c109783371d2939b234 ({username='slave'} at 127.20.218.66:32795)
I20240503 02:16:08.821313  4216 tablet_copy_service.cc:161] P 1d36807868fe4a82bca42c25831ce8a2: Beginning new tablet copy session on tablet 771a4190142b4fd5a9479417122c0881 from peer 6d54feb0bbc14c109783371d2939b234 at {username='slave'} at 127.20.218.66:32795: session id = 6d54feb0bbc14c109783371d2939b234-771a4190142b4fd5a9479417122c0881
I20240503 02:16:08.825932  4216 tablet_copy_source_session.cc:205] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:08.828419 12913 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:16:08.858522 12803 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:08.887225  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:08.896592 12803 tablet_bootstrap.cc:492] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Bootstrap complete.
I20240503 02:16:08.909231  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:08.922047  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:08.927532 12913 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 02:16:08.937801 12913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:08.959405 12803 ts_tablet_manager.cc:1367] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.551s	user 0.077s	sys 0.001s
I20240503 02:16:08.960230 12803 raft_consensus.cc:348] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:16:08.960474 12803 raft_consensus.cc:729] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:16:08.960728 12803 consensus_queue.cc:260] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:16:08.961372 12803 ts_tablet_manager.cc:1398] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:16:08.963155  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:08.969501  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 6d54feb0bbc14c109783371d2939b234-586ebf467f0f427fa62dee6223a39848 received from {username='slave'} at 127.20.218.66:33413
I20240503 02:16:08.969667  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 6d54feb0bbc14c109783371d2939b234-586ebf467f0f427fa62dee6223a39848 on tablet 586ebf467f0f427fa62dee6223a39848 with peer 6d54feb0bbc14c109783371d2939b234
I20240503 02:16:08.983712  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:08.988153 12913 tablet_copy_client.cc:783] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:08.988492 12913 tablet_copy_client.cc:647] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:08.989774  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:08.996748 12913 tablet_copy_client.cc:523] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:09.003669 12913 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:16:09.012140  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:09.041035  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:09.047091  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:09.062951  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:09.073474  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:09.102826  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:09.113198  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 66972us
I20240503 02:16:09.128757  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:09.192261 12913 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:09.192818 12913 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Bootstrap complete.
I20240503 02:16:09.193271 12913 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.190s	user 0.059s	sys 0.007s
I20240503 02:16:09.194325 12913 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:16:09.194568 12913 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:16:09.194950 12913 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:16:09.195856 12913 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:16:09.196725  4216 tablet_copy_service.cc:342] P 1d36807868fe4a82bca42c25831ce8a2: Request end of tablet copy session 6d54feb0bbc14c109783371d2939b234-771a4190142b4fd5a9479417122c0881 received from {username='slave'} at 127.20.218.66:32795
I20240503 02:16:09.196831  4216 tablet_copy_service.cc:433] P 1d36807868fe4a82bca42c25831ce8a2: ending tablet copy session 6d54feb0bbc14c109783371d2939b234-771a4190142b4fd5a9479417122c0881 on tablet 771a4190142b4fd5a9479417122c0881 with peer 6d54feb0bbc14c109783371d2939b234
W20240503 02:16:09.204249  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:09.215176  2412 raft_consensus.cc:1203] T 586ebf467f0f427fa62dee6223a39848 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:16:09.473105  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:09.477876  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:09.479760  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:09.490307  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:09.514600  2412 raft_consensus.cc:1203] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240503 02:16:09.534178  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:09.539423  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:09.558167  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:09.561439  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:09.583732  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:09.608225  4176 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:16:09.671152  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:09.702553  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:09.736964  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:09.751730  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:09.823973  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:10.041460  2280 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223"
dest_uuid: "08055479ea104aa290d19955d3720850"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33596
I20240503 02:16:10.041699  2280 raft_consensus.cc:595] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:16:10.114632  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:10.132023  2260 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:10.196816  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:10.218734  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:10.274336 13084 raft_consensus.cc:979] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: : Instructing follower 1d36807868fe4a82bca42c25831ce8a2 to start an election
I20240503 02:16:10.278813 13117 raft_consensus.cc:1067] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Signalling peer 1d36807868fe4a82bca42c25831ce8a2 to start an election
W20240503 02:16:10.291676  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:10.292611  4196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223"
dest_uuid: "1d36807868fe4a82bca42c25831ce8a2"
 from {username='slave'} at 127.20.218.65:34381
I20240503 02:16:10.292798  4196 raft_consensus.cc:482] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:16:10.292898  4196 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:10.296020  4196 raft_consensus.cc:504] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:10.297703  2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:16:10.297915  2412 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:10.300558  2412 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 2.
I20240503 02:16:10.301461  4129 leader_election.cc:302] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:16:10.301810 11817 raft_consensus.cc:2781] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:16:10.303059  4196 leader_election.cc:288] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:16:10.308974  2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51224b6355ed41dbb0703b7feb73c223" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "08055479ea104aa290d19955d3720850"
I20240503 02:16:10.309224  2280 raft_consensus.cc:3032] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:16:10.309283  2280 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Running, Role: LEADER
I20240503 02:16:10.309556  2280 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:10.309885  2280 raft_consensus.cc:3037] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:16:10.314429  2280 raft_consensus.cc:2454] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d36807868fe4a82bca42c25831ce8a2 in term 2.
I20240503 02:16:10.337288  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:10.360724 11817 raft_consensus.cc:686] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Becoming Leader. State: Replica: 1d36807868fe4a82bca42c25831ce8a2, State: Running, Role: LEADER
I20240503 02:16:10.361292 11817 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:10.379035  2133 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: term changed from 1 to 2, leader changed from 08055479ea104aa290d19955d3720850 (127.20.218.65) to 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69). New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:10.399829  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:10.407661  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:10.505506  4644 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:15:15.196029  4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:15.196895  4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:15.223798  4649 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:18.907474  4649 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:15:20.351425  5206 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 122920us
W20240503 02:15:21.044116  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.049311  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.049580  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.049810  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.050086  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.051613  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.051772  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.051949  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.052140  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.052480  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.054916  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.055244  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.055402  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.055620  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.055768  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:21.442332  4649 rebalancer_tool.cc:1084] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '1d36807868fe4a82bca42c25831ce8a2' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240503 02:15:21.442648  4649 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:23.509927  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:23.510100  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:23.510728  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:23.510902  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:23.511051  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:23.511253  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:23.511425  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:23.511540  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:23.511726  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:23.512051  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:23.512187  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:23.512305  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:23.512445  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:23.512590  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:23.512719  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:23.512837  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:23.731312  4649 rebalancer_tool.cc:1074] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:24.503931  4649 rebalancer_tool.cc:1074] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:25.056277  4649 rebalancer_tool.cc:1074] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:25.230280  4649 rebalancer_tool.cc:1084] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '408af0ab5520465e942cc714bfc6c73b' move ignored: Illegal state: tablet 02e0b954008c4f30af414da6cb5d59c2: replica 408af0ab5520465e942cc714bfc6c73b is already present
I20240503 02:15:25.230476  4649 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:27.726779  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 7fd07044862a481aa387698017467c01 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.727109  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.727313  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:27.727499  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:27.727587  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.727687  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:27.727777  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.727878  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.727967  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:27.728123  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.728211  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:28.509485  4649 rebalancer_tool.cc:1074] tablet ae239d73f0bc4d9aac025e40489e81a6: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:28.559398  4649 rebalancer_tool.cc:1074] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:28.777913  4649 rebalancer_tool.cc:1084] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move ignored: Illegal state: tablet 43e248f0cb764d229802851c4e73cc96: replica 1d36807868fe4a82bca42c25831ce8a2 is already present
I20240503 02:15:28.778064  4649 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:31.128520  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:31.128782  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:31.129622  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:31.129839  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:31.130113  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:31.590770  4649 rebalancer_tool.cc:1074] tablet 7a62b3ef44e94e92b0feed9c228f5702: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:31.655638  4649 rebalancer_tool.cc:1074] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:31.810995  4649 rebalancer_tool.cc:1074] tablet 1eeb69e5eb3c459ca749aa37672ec057: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:31.904213  4649 rebalancer_tool.cc:1084] tablet a7ab8b0c8db54565a6066b9a7d1569d8: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move ignored: Illegal state: tablet a7ab8b0c8db54565a6066b9a7d1569d8: replica 7fd07044862a481aa387698017467c01 is already present
I20240503 02:15:31.904392  4649 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:33.562314  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:33.562770  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:33.563575  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:33.564539  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:33.564909  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:33.565205  4649 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
I20240503 02:15:34.236912  4649 rebalancer_tool.cc:1074] tablet 1ec063c94fe941ebb1c866b5b8935ece: '08055479ea104aa290d19955d3720850' -> '' move scheduled
W20240503 02:15:35.602811  7762 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s	user 0.001s	sys 0.000s
I20240503 02:15:45.499500  4649 rebalancer_tool.cc:1123] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move is abandoned: Not found: Server with UUID 08055479ea104aa290d19955d3720850 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:45.584932  4649 rebalancer_tool.cc:1136] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:47.636107  4649 rebalancer_tool.cc:1136] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:15:50.657058  4649 rebalancer_tool.cc:1136] tablet 7a62b3ef44e94e92b0feed9c228f5702: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:50.782317  4649 rebalancer_tool.cc:1074] tablet 2de1d29dbea945eba58f8eadb019d766: '7fd07044862a481aa387698017467c01' -> '' move scheduled
I20240503 02:15:50.872205  4649 rebalancer_tool.cc:1074] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:51.375385  4649 rebalancer_tool.cc:1136] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:53.969025  4649 rebalancer_tool.cc:1136] tablet 1ec063c94fe941ebb1c866b5b8935ece: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:15:54.081110  4649 rebalancer_tool.cc:1136] tablet 1eeb69e5eb3c459ca749aa37672ec057: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:54.310020  4649 rebalancer_tool.cc:1136] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:54.484709  4649 rebalancer_tool.cc:1136] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:54.590189  4649 rebalancer_tool.cc:1136] tablet ae239d73f0bc4d9aac025e40489e81a6: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:54.797616  4649 rebalancer_tool.cc:1074] tablet 8870feaf47a64a929dd2e584ece47218: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:54.909687  4649 rebalancer_tool.cc:1074] tablet 62a3dcc2de594c8982477a0229f71191: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:55.113457  4649 rebalancer_tool.cc:1074] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:55.283854  4649 rebalancer_tool.cc:1136] tablet 62a3dcc2de594c8982477a0229f71191: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:55.374289  4649 rebalancer_tool.cc:1136] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:15:57.358402  4649 rebalancer_tool.cc:1136] tablet 2de1d29dbea945eba58f8eadb019d766: '7fd07044862a481aa387698017467c01' -> '' move completed: OK
I20240503 02:15:57.460466  4649 rebalancer_tool.cc:1074] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:57.592657  4649 rebalancer_tool.cc:1074] tablet 592cf4d5b35e49bfa4173343d64367f4: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:57.635607  4649 rebalancer_tool.cc:1074] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:04.319813  4649 rebalancer_tool.cc:1136] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:04.502305  4649 rebalancer_tool.cc:1136] tablet 8870feaf47a64a929dd2e584ece47218: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:04.595957  4649 rebalancer_tool.cc:1074] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:16:04.783360  4649 rebalancer_tool.cc:1074] tablet eadcb49d12664e6fb5a5ad8dbe47f342: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:07.269299  4649 rebalancer_tool.cc:1136] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:07.435304  4649 rebalancer_tool.cc:1136] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:07.530488  4649 rebalancer_tool.cc:1136] tablet 592cf4d5b35e49bfa4173343d64367f4: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:07.742267  4649 rebalancer_tool.cc:1074] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:07.877681  4649 rebalancer_tool.cc:1074] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:07.911342  4649 rebalancer_tool.cc:1136] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:07.980142  4649 rebalancer_tool.cc:1136] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:08.068982  4649 rebalancer_tool.cc:1136] tablet eadcb49d12664e6fb5a5ad8dbe47f342: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
Illegal state: tablet server 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): unacceptable health status UNAVAILABLE

W20240503 02:16:10.568758  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:16:10.612462  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.626137  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:10.640518  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:16:10.662964  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.665400  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:10.671348  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:10.707340  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:10.725978  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:10.738101  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:10.748268  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:10.755218  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:10.775846  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:10.787191  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:10.793567  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.801918  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:10.807566  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:10.808014  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:10.808404  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:16:10.813947  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.834470  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:10.861690  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.869429  2412 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:16:10.870118 11817 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:16:10.907284  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.910001  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:10.913378  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:16:10.937186  2280 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:16:10.938127 13210 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:16:11.037124  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:16:11.038309  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:16:11.055780  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:11.091282  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:11.150471  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:16:11.186343  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:16:11.191594  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:16:11.197928  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:11.199493  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:11.231506  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:11.235173  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:11.255015  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:11.258389  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:11.288076  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:11.312397  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:11.446311  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:11.459849  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:11.508049  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:11.526072  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:11.551575  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:11.554154  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:11.637064  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:11.655517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:11.982538  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:12.005265  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:12.015944  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:12.018741  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:12.051687  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:12.093894  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:12.102804  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:12.124822  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:12.165715  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 66090.6us
I20240503 02:16:12.272516  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:12.291389  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:12.325438  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:12.331059  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:12.351758  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:12.365070  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:12.368075  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:12.383723  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:12.393579  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:12.410341  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:12.416491  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:12.450083  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:12.485409  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:12.556141  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:12.677244  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:12.827001  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:12.846022  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:12.934291  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:12.942696  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:12.983951  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:12.984019  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:12.988741  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:12.997707  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.021229  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:13.052477  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 49744.9us
W20240503 02:16:13.186645  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:16:13.209261  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 55722.8us
W20240503 02:16:13.211439  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:13.213682  4196 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:16:13.225666  2412 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:16:13.226485 13208 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:16:13.241636 13208 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:13.246464  2410 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:13.256636  2135 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 5 to 7, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:16:13.295351  2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 51224b6355ed41dbb0703b7feb73c223 with delete_type TABLET_DATA_TOMBSTONED (TS 08055479ea104aa290d19955d3720850 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38192
I20240503 02:16:13.296962  4196 consensus_queue.cc:237] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
W20240503 02:16:13.301633  4132 consensus_peers.cc:485] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285): Couldn't send request to peer 408af0ab5520465e942cc714bfc6c73b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51224b6355ed41dbb0703b7feb73c223. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:13.304214  2412 raft_consensus.cc:1261] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:16:13.306638 13478 consensus_queue.cc:1035] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:16:13.310606 13478 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:13.318498  2412 raft_consensus.cc:2932] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:13.334448  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 51224b6355ed41dbb0703b7feb73c223 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:13.336182  2131 catalog_manager.cc:5484] T 51224b6355ed41dbb0703b7feb73c223 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 7 to 8, VOTER 408af0ab5520465e942cc714bfc6c73b (127.20.218.71) added. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:16:13.343305 13530 tablet_replica.cc:330] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: stopping tablet replica
I20240503 02:16:13.343641 13530 raft_consensus.cc:2229] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:13.343915 13530 raft_consensus.cc:2258] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:13.345099 13530 ts_tablet_manager.cc:1875] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:16:13.349946  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.367311 13530 ts_tablet_manager.cc:1888] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:16:13.367467 13530 log.cc:1196] T 51224b6355ed41dbb0703b7feb73c223 P 08055479ea104aa290d19955d3720850: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/51224b6355ed41dbb0703b7feb73c223
I20240503 02:16:13.368227  2127 catalog_manager.cc:4830] TS 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): tablet 51224b6355ed41dbb0703b7feb73c223 (table rebalance_test_table_2 [id=e6f074dfde9442ee808f069f0aa37634]) successfully deleted
W20240503 02:16:13.379609  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:16:13.398945  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.416529  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:13.416782  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.418071  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:13.420359  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:13.427582  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:13.456827  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:13.457885  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:16:13.472867  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.528256  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:13.535519  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.591739  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:13.716245  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:16:13.741756  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:16:13.770334 13588 ts_tablet_manager.cc:902] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Initiating tablet copy from peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:16:13.782990  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:13.831444 13588 tablet_copy_client.cc:308] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.69:39741
I20240503 02:16:13.836333  4216 tablet_copy_service.cc:140] P 1d36807868fe4a82bca42c25831ce8a2: Received BeginTabletCopySession request for tablet 51224b6355ed41dbb0703b7feb73c223 from peer 408af0ab5520465e942cc714bfc6c73b ({username='slave'} at 127.20.218.71:41757)
I20240503 02:16:13.836534  4216 tablet_copy_service.cc:161] P 1d36807868fe4a82bca42c25831ce8a2: Beginning new tablet copy session on tablet 51224b6355ed41dbb0703b7feb73c223 from peer 408af0ab5520465e942cc714bfc6c73b at {username='slave'} at 127.20.218.71:41757: session id = 408af0ab5520465e942cc714bfc6c73b-51224b6355ed41dbb0703b7feb73c223
I20240503 02:16:13.841435  4216 tablet_copy_source_session.cc:205] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:13.847450 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51224b6355ed41dbb0703b7feb73c223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:13.866086 13588 tablet_copy_client.cc:783] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:13.866547 13588 tablet_copy_client.cc:647] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:13.872128 13588 tablet_copy_client.cc:523] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:16:13.884780 13588 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap starting.
I20240503 02:16:13.977352  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:14.036656  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:14.041314  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:14.132632 13588 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:14.133093 13588 tablet_bootstrap.cc:492] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Bootstrap complete.
I20240503 02:16:14.133390 13588 ts_tablet_manager.cc:1367] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Time spent bootstrapping tablet: real 0.249s	user 0.072s	sys 0.005s
I20240503 02:16:14.191160 13588 raft_consensus.cc:348] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:14.191426 13588 raft_consensus.cc:729] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408af0ab5520465e942cc714bfc6c73b, State: Initialized, Role: FOLLOWER
I20240503 02:16:14.191738 13588 consensus_queue.cc:260] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:16:14.192966  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:14.215533 13588 ts_tablet_manager.cc:1398] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b: Time spent starting tablet: real 0.082s	user 0.003s	sys 0.013s
I20240503 02:16:14.217571  4216 tablet_copy_service.cc:342] P 1d36807868fe4a82bca42c25831ce8a2: Request end of tablet copy session 408af0ab5520465e942cc714bfc6c73b-51224b6355ed41dbb0703b7feb73c223 received from {username='slave'} at 127.20.218.71:41757
I20240503 02:16:14.217698  4216 tablet_copy_service.cc:433] P 1d36807868fe4a82bca42c25831ce8a2: ending tablet copy session 408af0ab5520465e942cc714bfc6c73b-51224b6355ed41dbb0703b7feb73c223 on tablet 51224b6355ed41dbb0703b7feb73c223 with peer 408af0ab5520465e942cc714bfc6c73b
I20240503 02:16:14.221457  4460 raft_consensus.cc:1203] T 51224b6355ed41dbb0703b7feb73c223 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:16:14.239567  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:14.269950  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:14.282948  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:14.297567  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:14.309321  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:14.323290  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:14.328128  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:14.362948  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:14.370815  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:14.374827  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:14.379189  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:14.420567  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:14.431456  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:14.443461  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:14.450546  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:14.486982  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:14.527469  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:14.532488  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:14.940081  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:15.024343  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:16:15.026898  2669 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:16:15.027628 13704 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:16:15.048476 13704 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:15.053923  2668 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:15.056808  2134 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 10 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:16:15.102150  2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a4190142b4fd5a9479417122c0881 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52816
I20240503 02:16:15.102312  4196 consensus_queue.cc:237] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:16:15.105933  2669 raft_consensus.cc:1261] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Refusing update from remote peer 1d36807868fe4a82bca42c25831ce8a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:16:15.107936 13478 consensus_queue.cc:1035] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 02:16:15.111537  4132 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:16:15.112767 13478 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:15.114663  2669 raft_consensus.cc:2932] T 771a4190142b4fd5a9479417122c0881 P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:15.118261  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:15.124048  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:15.129930  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 771a4190142b4fd5a9479417122c0881 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:16:15.133989  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:15.135779  2131 catalog_manager.cc:5484] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 11 to 12, VOTER 08055479ea104aa290d19955d3720850 (127.20.218.65) added. New cstate: current_term: 2 leader_uuid: "1d36807868fe4a82bca42c25831ce8a2" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:16:15.165854  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:15.190394 13767 tablet_replica.cc:330] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:16:15.192193 13767 raft_consensus.cc:2229] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:16:15.195289 13767 raft_consensus.cc:2258] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:16:15.198304 13767 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:16:15.221421  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:15.233289  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:15.253268 13767 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:16:15.257058 13767 log.cc:1196] T 771a4190142b4fd5a9479417122c0881 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/771a4190142b4fd5a9479417122c0881
I20240503 02:16:15.259618  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:15.263595  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 771a4190142b4fd5a9479417122c0881 (table rebalance_test_table_4 [id=884eb53e27af4de69b5921029b1ca76e]) successfully deleted
W20240503 02:16:15.399856  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:15.420914  4645 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:15:15.519881  4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:15.521438  4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:15.583326  4651 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:19.121755  4651 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:15:21.050190  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.050449  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.050846  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.051012  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.052637  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.052904  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.053052  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.053222  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.053373  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.053534  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.053730  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.053898  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.054116  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.054268  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
I20240503 02:15:21.950951  4651 rebalancer_tool.cc:1074] tablet 8870feaf47a64a929dd2e584ece47218: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:22.501482  4651 rebalancer_tool.cc:1074] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:23.226943  4651 rebalancer_tool.cc:1074] tablet f0a088d230d0462980648628eae548d2: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:23.694978  4651 rebalancer_tool.cc:1074] tablet eadcb49d12664e6fb5a5ad8dbe47f342: 'e9fdaf326bb64edabc43b366df870f83' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:24.168021  4651 rebalancer_tool.cc:1074] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:24.914511  4651 rebalancer_tool.cc:1074] tablet 9b822c691d824b2ba25e967535c9734e: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:25.331787  4651 rebalancer_tool.cc:1084] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 02:15:25.331992  4651 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:27.761320  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.775723  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.776734  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.782017  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:27.782920  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.783229  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.783342  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:27.783463  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:27.783586  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:28.065611  4651 rebalancer_tool.cc:1074] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:28.711884  4651 rebalancer_tool.cc:1074] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:29.374166  4651 rebalancer_tool.cc:1074] tablet 592cf4d5b35e49bfa4173343d64367f4: 'e9fdaf326bb64edabc43b366df870f83' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:29.426925  4651 rebalancer_tool.cc:1074] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:29.593258  4651 rebalancer_tool.cc:1074] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:29.857553  4651 rebalancer_tool.cc:1074] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move scheduled
I20240503 02:15:30.801070  4651 rebalancer_tool.cc:1084] tablet a7ab8b0c8db54565a6066b9a7d1569d8: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:30.801273  4651 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:32.439891  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.440397  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:32.442219  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:32.444871  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.445137  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:32.445405  4651 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:32.688019  4651 rebalancer_tool.cc:1074] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
W20240503 02:15:36.178957  7721 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 87584.4us
I20240503 02:15:43.041116  4651 rebalancer_tool.cc:1136] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:45.546695  4651 rebalancer_tool.cc:1136] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:58.053791  4651 rebalancer_tool.cc:1136] tablet 9b822c691d824b2ba25e967535c9734e: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:58.193104  4651 rebalancer_tool.cc:1074] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:58.281936  4651 rebalancer_tool.cc:1136] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:16:00.897534  4651 rebalancer_tool.cc:1136] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:00.983846  4651 rebalancer_tool.cc:1136] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:01.094035  4651 rebalancer_tool.cc:1136] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:01.298146  4651 rebalancer_tool.cc:1136] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:16:01.448383  4651 rebalancer_tool.cc:1136] tablet f0a088d230d0462980648628eae548d2: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:16:01.646186  4651 rebalancer_tool.cc:1136] tablet 8870feaf47a64a929dd2e584ece47218: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:16:01.719471  4651 rebalancer_tool.cc:1136] tablet eadcb49d12664e6fb5a5ad8dbe47f342: 'e9fdaf326bb64edabc43b366df870f83' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:16:01.920650  4651 rebalancer_tool.cc:1136] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:16:02.026335  4651 rebalancer_tool.cc:1136] tablet 592cf4d5b35e49bfa4173343d64367f4: 'e9fdaf326bb64edabc43b366df870f83' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:16:02.173738  4651 rebalancer_tool.cc:1074] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:02.326274  4651 rebalancer_tool.cc:1074] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:16:02.351107  4651 rebalancer_tool.cc:1074] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:02.430509  4651 rebalancer_tool.cc:1074] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:02.533059  4651 rebalancer_tool.cc:1136] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:02.638480  4651 rebalancer_tool.cc:1136] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:02.667440  4651 rebalancer_tool.cc:1136] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:02.819408  4651 rebalancer_tool.cc:1136] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:03.083060  4651 rebalancer_tool.cc:1136] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move completed: OK
I20240503 02:16:03.195760  4651 rebalancer_tool.cc:1074] tablet 771a4190142b4fd5a9479417122c0881: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:03.330027  4651 rebalancer_tool.cc:1074] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:16:03.451232  4651 rebalancer_tool.cc:1074] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:03.578387  4651 rebalancer_tool.cc:1074] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:03.676306  4651 rebalancer_tool.cc:1074] tablet 734027e195414b15811ac9d32b132db8: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:16:05.980232  4651 rebalancer_tool.cc:1136] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:06.146363  4651 rebalancer_tool.cc:1136] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:06.279947  4651 rebalancer_tool.cc:1136] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:08.305634  4651 rebalancer_tool.cc:1074] tablet 51224b6355ed41dbb0703b7feb73c223: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:11.213999  4651 rebalancer_tool.cc:1136] tablet 734027e195414b15811ac9d32b132db8: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:11.429693  4651 rebalancer_tool.cc:1136] tablet 771a4190142b4fd5a9479417122c0881: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:13.639878  4651 rebalancer_tool.cc:1136] tablet 51224b6355ed41dbb0703b7feb73c223: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
Illegal state: tablet server 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): unacceptable health status UNAVAILABLE

W20240503 02:16:15.460934  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:16:15.462283  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:15.559536  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:15.587850  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:15.622547  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:15.634491 13829 ts_tablet_manager.cc:902] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Initiating tablet copy from peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:16:15.651300  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:15.661911 13829 tablet_copy_client.cc:308] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.69:39741
I20240503 02:16:15.663120  4216 tablet_copy_service.cc:140] P 1d36807868fe4a82bca42c25831ce8a2: Received BeginTabletCopySession request for tablet 771a4190142b4fd5a9479417122c0881 from peer 08055479ea104aa290d19955d3720850 ({username='slave'} at 127.20.218.65:34381)
I20240503 02:16:15.663339  4216 tablet_copy_service.cc:161] P 1d36807868fe4a82bca42c25831ce8a2: Beginning new tablet copy session on tablet 771a4190142b4fd5a9479417122c0881 from peer 08055479ea104aa290d19955d3720850 at {username='slave'} at 127.20.218.65:34381: session id = 08055479ea104aa290d19955d3720850-771a4190142b4fd5a9479417122c0881
I20240503 02:16:15.671211  4216 tablet_copy_source_session.cc:205] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:16:15.677431 13829 ts_tablet_manager.cc:1875] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:16:15.692410 13829 ts_tablet_manager.cc:1888] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240503 02:16:15.705978 13829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a4190142b4fd5a9479417122c0881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:16:15.724345 13829 tablet_copy_client.cc:783] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet copy: Starting download of 0 data blocks...
I20240503 02:16:15.724862 13829 tablet_copy_client.cc:647] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:16:15.728121 13829 tablet_copy_client.cc:523] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:16:15.756098  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 51018.5us
W20240503 02:16:15.767935  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:15.776110 13829 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Bootstrap starting.
I20240503 02:16:15.801121  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:15.811307  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:15.814811  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:15.814939  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:15.839578  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:15.846633  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:15.862556  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:15.886621  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:15.901017  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:15.908181  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:16:15.922431  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:15.924383  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:15.946424  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:15.993849  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:16.001767  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:16.014495  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:16:16.020560  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:16.068521 13829 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:16:16.068974 13829 tablet_bootstrap.cc:492] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Bootstrap complete.
I20240503 02:16:16.069283 13829 ts_tablet_manager.cc:1367] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Time spent bootstrapping tablet: real 0.293s	user 0.053s	sys 0.002s
I20240503 02:16:16.070103 13829 raft_consensus.cc:348] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:16:16.070317 13829 raft_consensus.cc:729] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08055479ea104aa290d19955d3720850, State: Initialized, Role: FOLLOWER
I20240503 02:16:16.070580 13829 consensus_queue.cc:260] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
W20240503 02:16:16.071575  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:16:16.071640 13829 ts_tablet_manager.cc:1398] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
W20240503 02:16:16.072677  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:16.072703  4216 tablet_copy_service.cc:342] P 1d36807868fe4a82bca42c25831ce8a2: Request end of tablet copy session 08055479ea104aa290d19955d3720850-771a4190142b4fd5a9479417122c0881 received from {username='slave'} at 127.20.218.65:34381
I20240503 02:16:16.072804  4216 tablet_copy_service.cc:433] P 1d36807868fe4a82bca42c25831ce8a2: ending tablet copy session 08055479ea104aa290d19955d3720850-771a4190142b4fd5a9479417122c0881 on tablet 771a4190142b4fd5a9479417122c0881 with peer 08055479ea104aa290d19955d3720850
W20240503 02:16:16.179517  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:16.180387  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:16.222525  2280 raft_consensus.cc:1203] T 771a4190142b4fd5a9479417122c0881 P 08055479ea104aa290d19955d3720850 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
W20240503 02:16:16.277299  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:16.307420  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:16.320669  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:16.326139  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:16.370438  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:16.382251  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:16.449961  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:16.460100  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:16:16.549546  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:16.556214  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:16.571751  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:16.575678  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:16.583293  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:16.601318  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:16.607097  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:16.609622  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:16.611996  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:16.630859  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:16.637558  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:16.923563  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:16.968741  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:16.986706  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:17.014583  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:17.014725  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:17.062386  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:17.098052  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:17.102708  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:17.274233  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:17.279743  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:17.299245  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:17.340821  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:17.346271  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:17.351171  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:17.397115  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:17.444602  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:17.537308  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:17.558460  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:17.603040  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:17.606556  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:17.670125  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:17.700239  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:17.726493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:17.891834  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:17.931277  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:17.956036  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:17.989614  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:18.013775  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:18.032318  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:18.033622  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.038157  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:18.058336  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.065290  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:18.081998  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.094790  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:18.099316  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:18.099784  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:18.109704  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:18.139124  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:18.170333  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 48194.7us
I20240503 02:16:18.184698  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:18.207118  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:18.252107  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:18.327860  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:16:18.426029  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:16:18.428009  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:16:18.444353  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:18.457638  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:18.487793  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:18.529718  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:18.533696  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.546355  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:18.548538  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:16:18.556288  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.575613  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:18.575690  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.639238  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:18.660563  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:16:18.661861  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:18.743229  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:18.793776  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:18.838889  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:18.850471  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:18.882977  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:18.890311  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:18.897751  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:18.968075  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:19.084383  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:19.110486  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:19.137441  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:19.169356  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:19.169543  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:19.180164  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:19.205909  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:19.393873  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:19.416942  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:19.432037  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:19.462157  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:19.462831  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:19.469349  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:19.480222  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:19.488291  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:19.692826  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:19.699808  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:19.720553  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:19.725916  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:19.732038  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:19.747911  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:19.767349  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:19.972559  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:19.976011  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:20.002154  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:20.023989  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:20.031701  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:20.045424  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:20.054136  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:20.078480  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:20.225481  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:20.271155  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:20.299072  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:20.307754  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:20.315181  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:20.350845  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:20.351497  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:20.400647  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:16:20.454473  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:20.545235  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:20.555961  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:20.564733  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:20.580122  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:20.584964  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:20.610159  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:20.614904  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:16:20.617252  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:20.620335  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:20.682488  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:20.694566  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:20.756312  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:20.959539  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:20.978025  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:20.988256  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:20.996057  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:21.017980  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:21.021615  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:21.028180  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:21.039610  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:21.055019  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:21.063972  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:21.098699  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:21.148751  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:21.163204  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:21.163360  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:21.166678  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:21.192898  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:21.222241  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:21.258325  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:21.265358  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:16:21.266942  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:21.269464  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:21.297942  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:16:21.300343  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:21.311316  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:16:21.354202  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 53861.6us
W20240503 02:16:21.437837  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:16:21.464006  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:21.468271  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:21.486934  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:21.491680  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:21.542294  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:21.543334  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:16:21.553017  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:21.585822  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:21.706312  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:21.856535  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:21.887825  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:21.912936  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:21.978518  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:21.983246  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:22.015362  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:22.076370  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:22.080487  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:22.113780  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:22.138562  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:22.173445  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:22.224408  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:22.261605  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:22.282359  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:22.289397  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:22.398800  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:22.415189  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:22.426824  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:22.448845  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:22.452483  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:22.479339  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:22.530941  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:22.683508  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:22.743469  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:22.747596  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:22.762277  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:22.814675  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:22.849637  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:22.882673  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:22.959074  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:22.972270  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:22.981520  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:22.981607  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:22.993309  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:23.006636  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:23.013801  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:23.025646  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:23.051522  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:23.097205  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:23.104693  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:23.126966  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:23.194149  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:23.194283  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:23.211140  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:23.358356  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:23.362375  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:23.373548  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:23.416980  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:23.419512  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:23.437712  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:23.438642  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:23.457185  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:16:23.460168  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:23.474084  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:23.491940  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:23.618111  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:23.654356  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:23.792601  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:23.800393  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:23.805150  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:23.828985  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:23.878875  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:23.886843  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:23.940515  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:23.983001  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:24.021522  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:24.057075  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:24.070591  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:24.091583  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:24.103464  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:24.109547  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:24.153475  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 52081.5us
W20240503 02:16:24.178623  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:16:24.277228  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:24.338783  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:24.355669  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:24.362483  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:24.371814  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:24.394204  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:24.445498  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:24.450721  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:24.558068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:24.560562  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:24.592325  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:24.606403  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:24.630923  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:24.632344  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:24.646333  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:24.714339  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:24.853953  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:24.879158  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:24.903237  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:24.924052  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:24.946897  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:25.033567  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:25.073493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:25.196744  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:25.211241  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:25.257579  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:25.259688  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:25.275738  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:25.280625  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:25.282217  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:25.307677  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:25.333002  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:25.377758  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:25.517719  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:25.528817  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:25.543797  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:25.546993  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:25.639683  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:25.647542  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:25.671041  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:25.673813  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:25.683588  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:25.740641  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:25.768187  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:25.775987  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:25.786834  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:25.927438  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:25.984611  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:26.000064  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:26.014626  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:26.027006  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:26.054719  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:26.055850  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:26.073319  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:26.101655  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:26.107120  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:26.183801  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:16:26.213598  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:26.293387  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:26.306356  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:26.331652  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:26.381399  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:16:26.409026  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:16:26.422526  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:26.444335  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:26.457068  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:26.466051  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:26.500355  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:26.536552  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:26.543926  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:26.565146  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:26.568359  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:26.592813  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:26.607484  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:26.631253  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:26.661358  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:26.824568  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:26.897423  4129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 48456.3us
W20240503 02:16:26.980613  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:16:27.058126  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:27.058240  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:27.059407  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:27.066416  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:27.075191  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:27.091172  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:27.102828  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:27.103909  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:27.157797  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 54965us
W20240503 02:16:27.318226  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:27.395601  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:27.400825  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:27.422442  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:27.426692  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:27.439683  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:27.513872  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:27.564119  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:27.622018  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:27.642840  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:27.675060  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:27.707006  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:27.728324  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:27.741896  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:27.764113  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:27.786870  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:27.822199  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:27.840072  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:16:27.974436  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:27.982412  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:27.989787  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:28.018110  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:28.018666  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:28.020978  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:28.057488  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:28.058367  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:28.065126  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:28.115056  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:28.191169  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:16:28.244303  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:28.266103  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:28.274762  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:28.301841  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:28.314074  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:28.319411  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:28.338487  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:28.398784  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:28.406275  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:28.576416  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:28.598093  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:28.611439  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:28.615466  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:28.653424  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:28.661780  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:28.672982  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 64339.2us
W20240503 02:16:28.691887  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:28.705767  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:28.723865  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:16:28.724062  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:28.745198  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:16:28.840202  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:28.912118  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:28.923899  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:28.944962  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:28.969974  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:29.002475  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:29.034191  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:29.035768  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:29.200204  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:29.211722  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:29.222398  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:29.224256  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:29.242406  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:29.244190  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:29.254993  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:29.274752  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:29.469511  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:29.486726  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:29.513053  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:29.515281  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:29.538883  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:16:29.566004  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:29.588789  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:29.605008  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:29.773559  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:29.776878  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:29.781239  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:29.802444  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:29.816474  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:29.823545  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:29.916090  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:30.004601  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 63852us
I20240503 02:16:30.019416  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:30.025424  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:30.058809  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:30.060475  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:30.084388  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:30.137777  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 80016.6us
I20240503 02:16:30.150740  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:30.190641  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:30.202265  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:30.245440  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:16:30.303222  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:16:30.323362  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:30.350203  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:30.360608  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:30.366854  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:30.379902  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:30.381834  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:30.392369  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:30.392427  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:30.411077  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:30.461511  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:30.721450  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:30.734163  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:30.734174  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:30.790840  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:30.831897  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:30.836694  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:30.864667  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:30.872152  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:30.872958  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:30.875236  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:30.876927  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:30.917310  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:30.921289  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:30.950657  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:30.956847  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:30.957479  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 186127us
W20240503 02:16:30.967028  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:16:31.008842  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:31.067101  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:31.207624  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:31.249001  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:31.264366  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:31.281347  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:31.292865  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:31.294901  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:31.297439  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:16:31.308254  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:31.313998  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:31.316154  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:16:31.370330  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:31.375741  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:31.520020  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:31.531180  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:31.537763  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:16:31.587472  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:31.616482  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:31.618096  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:31.625777  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:31.632477  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:31.636832  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:31.713263  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:31.775118  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:16:31.871133  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:31.880784  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:31.903519  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:31.920696  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:31.942349  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:31.950043  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:31.967762  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:32.010701  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:32.141754  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:32.149792  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:32.168970  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:32.171294  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:32.213707  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:16:32.214576  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:16:32.256346  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:32.318030  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:32.332851  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:32.454988  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:32.458238  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:32.478046  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:32.501856  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:32.519351  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:32.534710  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:32.569911  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:32.590446  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:32.724206  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:32.781399  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:32.795648  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:32.812724  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:32.840140  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:32.866777  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:32.908072  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:32.931423  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:32.932479  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:33.048067  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:33.121388  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:33.148144  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:33.158365  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:33.169181  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:33.180624  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:33.199357  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:33.204352  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:33.223590  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:33.240058  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:33.246537  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:33.274600  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:33.300766  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:33.331036  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:33.356570  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:33.365959  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:33.391979  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:33.431367  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:33.470635  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:33.588578  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:33.654024  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:33.658208  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:33.664052  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:33.690124  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:33.690733  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:33.691454  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:33.695545  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:33.719966  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:33.726689  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:33.856428  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:33.869026  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:16:33.933768  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:33.990897  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:33.993266  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:34.017746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:34.029827  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:34.030678  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:34.046742  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:34.076299  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:34.142297  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:34.156931  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:34.160086  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:34.166256  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:34.185153  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:34.223313  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:34.232020  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:34.234961  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:34.499059  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:34.501173  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:34.512866  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:34.531158  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:34.531306  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:34.646260  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:34.646245  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:34.730264  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:34.871210  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:34.873018  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:34.891950  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:34.929081  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:34.936563  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:16:34.947522  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:34.993144  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:35.000555  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:35.143260  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:35.163957  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:35.189989  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:35.195986  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:35.205118  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:35.211935  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:35.229243  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:35.240165  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:35.257032  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:35.266145  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:16:35.300381  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:35.458549  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:35.486167  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:35.506525  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:35.519779  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:35.525444  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:35.535630  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:35.611173  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:35.622925  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:35.712268  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:35.980419  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:35.988840  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:36.003859  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 59691.1us
I20240503 02:16:36.020265  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:36.027071  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:36.067767  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:36.068106  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:36.089192  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:36.102649  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 108404us
I20240503 02:16:36.110939  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:36.137876  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:16:36.149988  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:36.150154  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:36.151824  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:36.152768  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:36.182647  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:36.223312  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:36.224390  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:36.239562  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:36.249203  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:36.255076  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:36.348973  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:16:36.352226  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:16:36.465000  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:36.479948  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:36.510962  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:36.511358  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:36.528865  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:36.555116  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:36.568413  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:36.599732  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:36.704334  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:36.750994  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:16:36.762265  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:36.763542  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:36.783530  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:36.792845  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:36.798282  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:36.805274  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 48357.5us
W20240503 02:16:36.807888  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:16:36.873793  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:36.917012  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:36.929697  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:36.939308  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:37.082255  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:37.096053  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:37.101606  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:37.113854  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:37.123795  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:37.128230  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:37.128779  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:37.169765  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:37.246917  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240503 02:16:37.263073  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:16:37.419518  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:37.455044  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:37.457212  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:37.463339  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:37.512845  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:37.527999  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:37.531550  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:37.636994  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:37.657236  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:37.680702  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:37.688944  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:37.701874  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:37.711943  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:37.724893  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:37.776993  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:37.798811  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:37.910722  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:16:37.976400  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:38.035038  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:38.041985  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:38.076943  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:38.120626  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:38.137977  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:38.170441  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:38.175724  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:38.180119  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:38.386124  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:38.412461  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:38.451987  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:38.454910  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:38.491667  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:38.530066  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:38.544237  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:38.553457  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:38.574231  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:38.598696  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:38.602717  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:38.630848  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:38.631490  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:38.672587  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:38.708833  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:38.712944  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:38.715346  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:16:38.723825  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:16:38.778327  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:16:38.782964  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:16:38.794142  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:38.847484  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:38.961504  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:38.961746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:38.967700  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:38.974449  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:38.996151  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:39.008139  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:39.015471  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:39.170547  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:39.192749  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:39.291954  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:39.308806  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:39.329845  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:39.347568  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:39.422350  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:39.431341  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:39.458642  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:39.462697  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:39.464296  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:39.472746  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:39.484696  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:39.522588  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:39.576731  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:39.584484  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:39.590726  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:39.857779  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:39.864292  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:39.875057  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:16:39.887037  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:39.899966  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:39.910563  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:39.934027  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:39.963574  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:40.051563  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:40.071005  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:40.095891  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:40.117051  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:40.125540  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:40.132872  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:40.141264  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:40.167632  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:40.206140  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:40.354071  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:40.359813  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:40.385097  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:40.402817  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:40.413389  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:40.417616  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:40.422106  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:40.468632  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:40.470757  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:40.554884  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:40.696164  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:40.719290  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:40.729072  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:40.735821  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:40.761767  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:40.780472  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:40.805779  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:40.823628  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:16:40.872427  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:40.928684  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:40.934312  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:40.943890  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:40.948951  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:40.980057  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:40.981174  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:40.985900  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:41.010288  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:41.058955  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:41.084663  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:16:41.101580  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:16:41.133148  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:41.184237  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:41.189989  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:41.206163  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:41.232156  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:41.235539  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:41.240123  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:41.273698  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:41.369601  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:16:41.413763  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:41.509688  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:41.521315  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:41.537405  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:41.609860  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:41.622520  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:41.638240  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:41.668226  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:41.682792  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:41.711510  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:41.719810  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:41.719877  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:41.748514  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:41.750695  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:41.753213  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:16:41.765753  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:41.820314  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:16:41.829442  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:41.905917  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:42.023042  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:42.076766  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:42.097855  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:42.099236  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:42.113946  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:42.121708  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:42.142006  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:42.146095  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:42.193776  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:42.288262  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:16:42.372411  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:42.396557  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:42.426561  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:42.431550  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:42.447811  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:42.453722  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:42.479822  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:42.504086  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:42.611263  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:42.666198  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:42.687772  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:42.690418  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:42.743883  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:42.765117  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:42.769377  2346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 58390.4us
I20240503 02:16:42.834544  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:42.863440  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:42.870705  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:42.927268  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:42.988271  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:43.092139  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:43.096771  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:43.112607  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:43.179333  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:43.179512  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:43.187765  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:43.195159  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:43.301121  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:43.315433  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:43.326346  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:43.373809  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:43.409307  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:43.414983  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:43.421185  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:43.429833  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:43.472223  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:43.487816  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:43.503825  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:43.520797  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:43.555366  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:43.591130  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:43.593482  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:43.613101  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:43.620636  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:43.621201  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:43.649870  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:43.675673  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:43.698410  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:43.727998  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:43.795225  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:43.850929  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:16:43.980129  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:43.988368  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:43.998009  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:44.002077  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:44.008378  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:44.038813  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:44.053267  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:44.053606  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:44.317505  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:44.329694  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:44.343851  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:44.351809  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:44.385353  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:44.459403  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:44.514711  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:44.530934  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:44.539460  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:44.578691  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:44.601614  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:44.629585  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:44.631662  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:44.690505  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:44.712365  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:44.863385  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 46704.3us
W20240503 02:16:44.869784  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 54925.1us
I20240503 02:16:44.975394  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:44.988267  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:45.003448  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:45.056793  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:45.059031  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:45.075034  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:16:45.077265  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:45.137357  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:45.141600  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:45.180555  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:45.215313  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:45.350716  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:45.414351  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:45.416857  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:45.434792  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:45.457401  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:45.477491  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:45.492601  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:45.518911  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:45.521530  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 128961us
I20240503 02:16:45.548255  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:45.551311  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:45.564774  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:45.575642  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:45.591802  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:45.608281  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:45.644899  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:45.648699  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:45.702720  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:45.767926  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:45.847424  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:45.902556  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:45.951797  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:45.952585  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:45.982302  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:45.989274  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:45.995532  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:45.995983  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:46.003953  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:46.033952  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:46.065424  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:46.148434  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:46.290457  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:46.309716  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:46.350711  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:46.366134  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:46.371534  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:46.406306  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:46.426656  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:46.427407  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:16:46.474328  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:46.501068  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:46.511736  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:46.566565  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:46.580972  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:46.637424  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:46.676326  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:46.679373  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:46.706714  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:46.875622  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 02:16:46.906375  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:46.917052  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:46.917274  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:16:46.919773  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:46.938736  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:46.963171  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:16:46.972339  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:46.978322  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:47.033205  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:47.037710  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:47.168676  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:47.173446  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:47.174296  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:47.198964  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:47.220113  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:47.287859  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:47.307157  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:47.365720  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:16:47.457480  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:47.483078  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:47.490012  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:47.495292  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:47.507637  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:47.510337  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:47.529407  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:47.633051  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 50086.3us
I20240503 02:16:47.641283  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:47.643585  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:47.644949  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:47.840041  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:47.850428  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:47.899565  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:47.905632  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:47.907447  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:47.910050  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:47.915426  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:47.917146  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:47.927536  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:47.952539  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:47.964219  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 56776.2us
I20240503 02:16:47.974511  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:48.155440  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:48.182861  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:48.184514  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:48.214879  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:48.245663  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:48.279664  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:48.291414  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:48.307027  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:48.380985  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:48.428707  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:48.440968  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:48.466387  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:48.489964  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:48.561661  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:48.577167  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:48.594633  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:48.596678  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:48.601142  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:48.636117  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:48.655794  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:48.704128  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:48.788887  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:16:48.878548  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:48.937301  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:48.960608  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:48.987251  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:48.988714  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:48.994432  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:49.005295  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:49.036593  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:49.053848  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:49.168197  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:49.171869  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:49.193557  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:49.197571  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:49.221624  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:49.225767  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:49.257563  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:49.447520  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:49.542232  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:49.556668  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:49.610316  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:49.618254  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:49.699625  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:49.699976  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:49.798246  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:49.884457  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:16:49.900198  2129 maintenance_manager.cc:419] P 6fad05a27a164335b985e3009b73dd44: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033396
W20240503 02:16:49.985193  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:49.995605  2128 maintenance_manager.cc:643] P 6fad05a27a164335b985e3009b73dd44: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.095s	user 0.043s	sys 0.006s Metrics: {"bytes_written":20387,"cfile_init":1,"dirs.queue_time_us":8509,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":7071,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1605,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":16654,"threads_started":2,"wal-append.queue_time_us":7790}
I20240503 02:16:49.996991  2129 maintenance_manager.cc:419] P 6fad05a27a164335b985e3009b73dd44: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7337 bytes on disk
I20240503 02:16:49.997900  2128 maintenance_manager.cc:643] P 6fad05a27a164335b985e3009b73dd44: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 02:16:50.059590  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:50.074721  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:50.075402  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:50.097949  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:50.103035  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:50.104290  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:50.133026  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:50.134215  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:50.317225  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:50.371258  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:50.391460  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:50.402725  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:50.417279  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:50.433475  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:50.435112  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:50.437834  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:50.473609  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 59786.7us
W20240503 02:16:50.474618  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:16:50.477869  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:50.696430  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:50.707062  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:50.758921  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:50.761538  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:50.770767  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:50.779726  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:50.785151  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:50.797472  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:50.879204  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:16:50.910679  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:50.918924  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:50.936069  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:50.936149  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:50.942885  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:50.946991  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:50.965770  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:50.969700  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:50.973078  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:50.993726  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:51.122761  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:16:51.189652  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:51.243561  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:51.251701  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:51.254824  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:51.262751  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:51.263717  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:51.276396  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:51.281025  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:51.289989  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:51.325630  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:16:51.369033  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:51.607775  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:51.656997  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:51.664135  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:51.675555  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:51.706364  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:51.710776  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:51.742749  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:51.746984  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:51.798086  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:51.820588  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:51.826910  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:51.853650  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:51.872213  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:51.908682  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240503 02:16:51.915015  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:51.920908  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:52.059890  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:16:52.177227  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:52.196004  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:16:52.204243  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:52.205511  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:52.231587  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:52.242686  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:52.263932  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:52.268934  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:16:52.280391  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:52.287048  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:52.327601  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:52.345077  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:52.383908  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:52.395941  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:52.404210  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:52.458354  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:52.466583  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:52.503403  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:52.596601  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:52.674566  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:52.692003  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:52.696406  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:52.697085  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:52.701949  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:52.711874  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:52.738126  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:52.794413  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:52.948204  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:52.995899  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:53.062004  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:53.080293  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:53.103251  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:53.138510  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:53.183622  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:53.193447  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 103417us
I20240503 02:16:53.205169  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:53.213241  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:53.227824  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:16:53.242269  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:53.243721  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:53.280007  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:53.282742  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:53.308524  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:53.340710  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:53.363397  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:53.380036  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:53.387414  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:53.496748  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:53.506553  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:53.544986  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:53.657474  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:53.660634  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:53.669222  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:53.689348  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:53.697372  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:53.702193  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:53.711584  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:53.741068  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:53.751523  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:53.786635  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:53.816644  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:53.826954  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:16:53.827100  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:53.842525  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:53.865056  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:53.868042  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:53.870967  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:53.900640  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:53.952510  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:54.125180  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:54.140228  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:54.141921  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:54.185148  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:54.186764  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:54.215076  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:54.216159  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:54.337713  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:54.508078  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:54.549360  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:54.566388  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:54.592227  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:54.618643  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:54.642194  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:54.684136  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:54.713379  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:54.740105  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:54.758155  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:54.760207  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:54.797457  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:54.798194  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:16:54.811146  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:54.840217  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:54.923216  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:55.063431  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:55.127697  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:55.127907  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:55.148631  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:55.149703  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:55.185222  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:55.204685  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:55.216598  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:55.392635  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:55.425859  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:55.437754  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:55.457938  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:55.478816  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:55.499037  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:55.536080  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:55.602196  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:55.683682  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:55.714768  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:16:55.896190  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:55.903389  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:55.929929  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:55.943966  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:55.946663  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:55.954669  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:16:55.969221  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.004106  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:56.013016  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:16:56.014786  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.029810  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:56.030692  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:56.043289  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:56.046473  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.073918  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:56.076264  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.084200  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:56.092386  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:16:56.102967  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.129317  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:56.217417  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:56.382444  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:16:56.419574  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:56.419713  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:56.432787  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:56.469630  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:56.471004  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:56.485544  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:56.493371  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:56.531632  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:56.595186  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:16:56.906179  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:56.926605  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:16:56.993352  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:57.012571  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:57.023633  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:57.054406  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:57.056455  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:57.080210  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:57.092495  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:57.110976  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:57.123916  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:57.180831  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:57.191305  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:57.198911  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:57.199316  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:57.238664  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:57.241472  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:57.256444  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:57.262140  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:57.301677  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:57.353248  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:16:57.357136  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:16:57.358559  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:57.358825  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 73021.6us
W20240503 02:16:57.360742  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:57.380395  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:57.418576  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:57.447954  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:57.489806  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:57.651099  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:16:57.939060  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.001844  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:58.027946  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:58.029302  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:58.031600  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:58.033718  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.041800  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:58.047355  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:16:58.048086  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:58.076009  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:58.212443  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:58.218356  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:58.262512  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:58.278879  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:58.314172  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:58.323424  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:58.380666  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:58.447778  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.474335  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(190ebb1335d64b279a41fa38feff22b4): perf score=0.033365
W20240503 02:16:58.482311  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:16:58.484530  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.484624  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(734027e195414b15811ac9d32b132db8): perf score=0.033363
I20240503 02:16:58.503080  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(8870feaf47a64a929dd2e584ece47218): perf score=0.033381
W20240503 02:16:58.522104  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.524847  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(190ebb1335d64b279a41fa38feff22b4): perf score=0.033375
I20240503 02:16:58.646294  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:58.647781  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:16:58.651439  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.663370  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:58.669135  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:58.672842  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:58.693717  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:16:58.693861  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:16:58.695112  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.698791  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:58.704528  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:58.909096  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.423s	user 0.081s	sys 0.042s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":59758,"dirs.queue_time_us":11543,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":7077,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":957,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141998,"threads_started":3,"wal-append.queue_time_us":68684}
I20240503 02:16:58.911994  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8): 4583 bytes on disk
I20240503 02:16:58.919183  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.007s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":176,"lbm_reads_lt_1ms":4}
W20240503 02:16:58.923110  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:58.993067  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:58.994628  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:59.004607  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.526s	user 0.074s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":479,"dirs.queue_time_us":72295,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":2406,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2983,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":144218,"threads_started":3,"wal-append.queue_time_us":15876}
I20240503 02:16:59.038491  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4): 4838 bytes on disk
I20240503 02:16:59.038821  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:59.039841  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:16:59.077656  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:16:59.087651  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:16:59.102591  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:59.106884  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:16:59.120325  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.616s	user 0.065s	sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":3779,"dirs.queue_time_us":29991,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":13436,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3404,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":91588,"threads_started":3,"wal-append.queue_time_us":34277}
I20240503 02:16:59.137611  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218): 4923 bytes on disk
I20240503 02:16:59.138684  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240503 02:16:59.142031  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:59.168468  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.642s	user 0.070s	sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":11478,"dirs.queue_time_us":34739,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":3148,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1289,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":60131,"threads_started":3,"wal-append.queue_time_us":10162}
I20240503 02:16:59.182453  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4): 4838 bytes on disk
I20240503 02:16:59.183878  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240503 02:16:59.197942  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:59.645323  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(f084f1783d4d477e9c3791984e681f4c): perf score=0.033361
I20240503 02:16:59.676168  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(9253de3f798f4a208d566765ed2ba54d): perf score=0.033380
I20240503 02:16:59.683115  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:16:59.687167  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(f084f1783d4d477e9c3791984e681f4c): perf score=0.033383
I20240503 02:16:59.688170  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:16:59.714978  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:16:59.727265  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:16:59.762513  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:16:59.779765  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:16:59.785092  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:16:59.797217  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(9253de3f798f4a208d566765ed2ba54d): perf score=0.033399
I20240503 02:16:59.853473  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:16:59.864164  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:16:59.942706  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:16:59.967850  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.291s	user 0.055s	sys 0.026s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9733,"dirs.run_cpu_time_us":334,"dirs.run_wall_time_us":12742,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1120,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34317,"threads_started":2,"wal-append.queue_time_us":19786}
I20240503 02:16:59.969348  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d): 4836 bytes on disk
I20240503 02:16:59.969980  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 02:17:00.077172  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.432s	user 0.075s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16102,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":8513,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8379,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":39738,"threads_started":2,"wal-append.queue_time_us":22583}
I20240503 02:17:00.078701  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c): 4585 bytes on disk
I20240503 02:17:00.079597  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 02:17:00.094600  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.407s	user 0.075s	sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21225,"dirs.run_cpu_time_us":278,"dirs.run_wall_time_us":6373,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":880,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31540,"threads_started":2,"wal-append.queue_time_us":10196}
I20240503 02:17:00.096132  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(1eeb69e5eb3c459ca749aa37672ec057): perf score=0.033488
W20240503 02:17:00.166316  2346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 52062.5us
W20240503 02:17:00.173215  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:00.198434  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:00.213173  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:00.215415  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:00.233222  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:00.237080  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.440s	user 0.068s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":28024,"dirs.run_cpu_time_us":329,"dirs.run_wall_time_us":11322,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":50897,"threads_started":2,"wal-append.queue_time_us":22319}
I20240503 02:17:00.242738  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d): 4836 bytes on disk
I20240503 02:17:00.243811  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:00.243687  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240503 02:17:00.307673  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(1eeb69e5eb3c459ca749aa37672ec057) complete. Timing: real 0.211s	user 0.056s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1591,"drs_written":1,"lbm_read_time_us":2004,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":4821,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26847,"threads_started":1,"wal-append.queue_time_us":19486}
I20240503 02:17:00.310075  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057): 4922 bytes on disk
I20240503 02:17:00.311857  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":212,"lbm_reads_lt_1ms":4}
I20240503 02:17:00.314218  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c): 4585 bytes on disk
I20240503 02:17:00.315761  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
W20240503 02:17:00.335067  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:00.460418  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:00.464444  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:00.518960  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:00.656952  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:00.810743  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:00.821926  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:00.875911  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:00.878202  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:00.898766  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:00.899137  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:00.917969  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:01.013065  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:01.036098  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:01.052910  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:01.123032  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:01.148375  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:01.152550  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:01.175516  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:01.206617  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:01.218735  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:01.249120  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:01.288583  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:01.342658  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:01.345197  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:01.391198  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:01.543908  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:01.552186  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:01.558135  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:01.562520  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:01.579929  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:01.592139  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:01.612360  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:01.628569  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:01.652310  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:17:01.674706  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:01.755759  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(d2351807a2ed47419f8d674fbeb9d0ec): perf score=0.033334
I20240503 02:17:01.827678  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(0137e646f1bd452b950d870107afa060): perf score=0.033358
I20240503 02:17:01.839638  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(51224b6355ed41dbb0703b7feb73c223): perf score=0.033352
I20240503 02:17:01.875335  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(d2351807a2ed47419f8d674fbeb9d0ec) complete. Timing: real 0.119s	user 0.053s	sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15765,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":2564,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1293,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":45087,"threads_started":2,"wal-append.queue_time_us":6692}
I20240503 02:17:01.876550  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec): 4583 bytes on disk
I20240503 02:17:01.877200  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 02:17:01.951781  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.124s	user 0.033s	sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4716,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":3361,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":883,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22787,"threads_started":2,"wal-append.queue_time_us":18104}
I20240503 02:17:01.953485  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060): 4923 bytes on disk
I20240503 02:17:01.957762  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":199,"lbm_reads_lt_1ms":4}
I20240503 02:17:01.982707  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.143s	user 0.058s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12922,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":6924,"drs_written":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1043,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24564,"threads_started":2,"wal-append.queue_time_us":11523}
I20240503 02:17:01.986912  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223): 4838 bytes on disk
I20240503 02:17:01.987663  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240503 02:17:02.033615  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:02.043517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:02.052162  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:02.056991  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:02.061358  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:02.070560  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:17:02.098387  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:02.113627  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:02.115069  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:02.196518  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:17:02.322158  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:02.342082  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:02.365559  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:02.372265  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:02.390535  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:02.393556  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:17:02.398752  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:02.427186  2609 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 59713.1us
I20240503 02:17:02.439692  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:02.457751  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:02.470943  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:02.471400  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:02.471508  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:02.485203  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:02.488327  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:02.490680  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:02.521488  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:17:02.579360  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:02.628669  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:02.822876  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:02.842234  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:02.860958  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:02.863464  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:02.924857  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:02.930603  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:02.936679  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:02.941862  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:03.003006  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:03.081184  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:03.082407  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:03.133076  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(75ba33132e784f408f3e26882aea3d15): perf score=0.033362
I20240503 02:17:03.201378  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.068s	user 0.039s	sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":664,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1724,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1090,"threads_started":2,"wal-append.queue_time_us":404}
I20240503 02:17:03.206209  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15): 4923 bytes on disk
I20240503 02:17:03.208669  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":161,"lbm_reads_lt_1ms":4}
I20240503 02:17:03.217878  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702): perf score=0.033387
I20240503 02:17:03.236254  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702): perf score=0.033386
I20240503 02:17:03.245635  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(75ba33132e784f408f3e26882aea3d15): perf score=0.033397
I20240503 02:17:03.256436  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:03.278074  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:03.342186  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:03.354054  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.117s	user 0.047s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16855,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":4104,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1109,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25534,"threads_started":2,"wal-append.queue_time_us":8599}
I20240503 02:17:03.355935  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702): 4838 bytes on disk
I20240503 02:17:03.364611  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 02:17:03.402117  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:03.426597  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.181s	user 0.070s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":26717,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":4910,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5183,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":33119,"threads_started":2,"wal-append.queue_time_us":6364}
I20240503 02:17:03.431247  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15): 4923 bytes on disk
I20240503 02:17:03.431921  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 02:17:03.433832  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(9b822c691d824b2ba25e967535c9734e): perf score=0.033445
I20240503 02:17:03.446406  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:03.454008  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:03.498873  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:03.560187  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:03.577558  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:03.577247  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:03.584365  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:03.598078  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:03.601555  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:03.618912  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.185s	user 0.045s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1553,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1248,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17690,"threads_started":1,"wal-append.queue_time_us":15540}
I20240503 02:17:03.620321  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e): 4583 bytes on disk
I20240503 02:17:03.624286  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 02:17:03.632977  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.415s	user 0.051s	sys 0.034s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":30041,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":51933,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2476,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":175378,"threads_started":2,"wal-append.queue_time_us":144050}
I20240503 02:17:03.639017  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702): 4838 bytes on disk
I20240503 02:17:03.640154  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4}
I20240503 02:17:03.647532  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:03.664209  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:03.675268  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:03.734211  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:17:03.749002  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:03.766887  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:03.778165  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:03.790277  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:03.864769  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:17:03.946255  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:04.016736  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:04.038367  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:04.049970  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:04.093540  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:04.096414  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:04.128176  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(a7ab8b0c8db54565a6066b9a7d1569d8): perf score=0.033339
I20240503 02:17:04.132140  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:04.139617  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:04.144878  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(43e248f0cb764d229802851c4e73cc96): perf score=0.033341
W20240503 02:17:04.201254  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:04.215663  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(43e248f0cb764d229802851c4e73cc96): perf score=0.033354
W20240503 02:17:04.240397  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:04.289141  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.144s	user 0.065s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":19213,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":6036,"drs_written":1,"lbm_read_time_us":1686,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":922,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":22653,"threads_started":2,"wal-append.queue_time_us":3536}
I20240503 02:17:04.290943  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96): 4838 bytes on disk
I20240503 02:17:04.291724  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":146,"lbm_reads_lt_1ms":4}
I20240503 02:17:04.326097  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:04.333783  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:04.368669  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.153s	user 0.043s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2949,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":49963,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10354,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":31367,"threads_started":2,"wal-append.queue_time_us":21412}
I20240503 02:17:04.370184  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96): 4838 bytes on disk
I20240503 02:17:04.371078  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 02:17:04.371253  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(a7ab8b0c8db54565a6066b9a7d1569d8): perf score=0.033401
I20240503 02:17:04.371621  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:04.379686  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:04.405609  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(a7ab8b0c8db54565a6066b9a7d1569d8) complete. Timing: real 0.277s	user 0.081s	sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10084,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":3194,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1191,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":22826,"threads_started":2,"wal-append.queue_time_us":14876}
I20240503 02:17:04.407850  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8): 4923 bytes on disk
I20240503 02:17:04.409200  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
I20240503 02:17:04.412861  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:04.450320  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(a7ab8b0c8db54565a6066b9a7d1569d8) complete. Timing: real 0.079s	user 0.042s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11041,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":14928,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1003,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11196,"threads_started":2,"wal-append.queue_time_us":433}
I20240503 02:17:04.451786  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8): 4923 bytes on disk
I20240503 02:17:04.452453  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 02:17:04.498528  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:04.499205  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:04.534544  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:04.646435  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:17:04.741068  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:04.769610  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:04.771037  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:04.797410  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:04.798652  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:04.821715  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:04.843351  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:04.853603  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:05.081084  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:05.081290  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:05.082159  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:05.098613  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:05.099658  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:05.101194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:05.106364  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:05.114907  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:05.298590  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2): perf score=0.033343
I20240503 02:17:05.301645  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:05.307242  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:05.311810  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:05.337111  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:05.350531  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:05.373220  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:05.376627  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2): perf score=0.033361
I20240503 02:17:05.394191  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.095s	user 0.043s	sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4613,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":4919,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9007,"threads_started":2,"wal-append.queue_time_us":4342}
I20240503 02:17:05.395222  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:05.396087  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2): 4583 bytes on disk
I20240503 02:17:05.396684  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 02:17:05.415068  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0): perf score=0.033382
I20240503 02:17:05.445641  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:05.457881  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(586ebf467f0f427fa62dee6223a39848): perf score=0.033385
I20240503 02:17:05.572896  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.196s	user 0.036s	sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":21308,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":2832,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":861,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":3068288,"thread_start_us":49768,"threads_started":2,"wal-append.queue_time_us":25612}
I20240503 02:17:05.575627  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2): 4583 bytes on disk
I20240503 02:17:05.576701  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
I20240503 02:17:05.582687  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.125s	user 0.055s	sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2578,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":2370,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":975,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3090,"threads_started":2,"wal-append.queue_time_us":393}
I20240503 02:17:05.584715  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848): 4923 bytes on disk
I20240503 02:17:05.585572  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
I20240503 02:17:05.648097  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.233s	user 0.042s	sys 0.032s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12887,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":14146,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":990,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13413,"threads_started":2,"wal-append.queue_time_us":502}
I20240503 02:17:05.651180  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0): 4838 bytes on disk
I20240503 02:17:05.652112  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 02:17:05.690383  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:05.702579  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:05.719061  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:05.721823  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:05.745384  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:05.784327  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:05.836166  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:05.874365  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:05.945756  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:05.984220  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 142271us
W20240503 02:17:05.997077  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:06.028674  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:06.044435  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:06.118628  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:06.159272  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:06.165514  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:06.189730  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:06.194790  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:06.266904  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:06.293721  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:06.304319  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:06.309619  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:06.319620  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:06.321827  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:06.328917  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:06.386330  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:06.404189  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:06.458438  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:06.476289  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:06.502928  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:06.552767  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:06.556255  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:17:06.585153  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1): perf score=0.033365
I20240503 02:17:06.592198  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(02e0b954008c4f30af414da6cb5d59c2): perf score=0.033356
W20240503 02:17:06.595665  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:06.654546  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1): perf score=0.033375
I20240503 02:17:06.697376  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(02e0b954008c4f30af414da6cb5d59c2) complete. Timing: real 0.105s	user 0.068s	sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15880,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":1842,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":893,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":32390,"threads_started":2,"wal-append.queue_time_us":12143}
I20240503 02:17:06.698820  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2): 4838 bytes on disk
I20240503 02:17:06.700727  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 02:17:06.714494  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.129s	user 0.054s	sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":402,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1706,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":814,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6329,"threads_started":2,"wal-append.queue_time_us":362}
I20240503 02:17:06.715548  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.061s	user 0.039s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6713,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":8270,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2541,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7116,"threads_started":2,"wal-append.queue_time_us":339}
I20240503 02:17:06.716347  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1): 4583 bytes on disk
I20240503 02:17:06.717139  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1): 4583 bytes on disk
I20240503 02:17:06.717430  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 02:17:06.719703  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 02:17:06.870151  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:06.879699  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:06.880383  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:06.903743  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:06.912263  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:06.959533  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:06.966480  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:07.013137  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:07.015767  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 50529.5us
W20240503 02:17:07.027915  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:17:07.122660  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:07.199591  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:07.247555  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:07.267107  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:07.270428  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:07.299304  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:07.307269  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:07.347005  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:07.398823  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:07.400576  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:07.403690  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:07.426013  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:07.437702  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:07.447600  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:07.511133  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:07.516453  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:07.530071  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:07.595045  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:07.664705  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab): perf score=0.033354
W20240503 02:17:07.677289  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:17:07.706203  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab): perf score=0.033358
I20240503 02:17:07.724745  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6): perf score=0.033359
I20240503 02:17:07.746786  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.082s	user 0.066s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":4584,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6802,"threads_started":2,"wal-append.queue_time_us":6393}
I20240503 02:17:07.754581  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(592cf4d5b35e49bfa4173343d64367f4): perf score=0.033373
W20240503 02:17:07.822003  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:07.830762  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.124s	user 0.061s	sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23015,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":3604,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4148,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":33351,"threads_started":2,"wal-append.queue_time_us":7015}
I20240503 02:17:07.832875  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab): 4923 bytes on disk
I20240503 02:17:07.833912  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4}
I20240503 02:17:07.866312  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(592cf4d5b35e49bfa4173343d64367f4) complete. Timing: real 0.111s	user 0.036s	sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1654,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14567,"threads_started":1,"wal-append.queue_time_us":8089}
I20240503 02:17:07.867895  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab): 4923 bytes on disk
I20240503 02:17:07.874008  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 02:17:07.875193  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4): 4583 bytes on disk
I20240503 02:17:07.877734  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240503 02:17:08.039459  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.314s	user 0.091s	sys 0.065s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":23215,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":30939,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3553,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":896,"thread_start_us":112377,"threads_started":2,"wal-append.queue_time_us":89397}
I20240503 02:17:08.041399  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6): 4838 bytes on disk
I20240503 02:17:08.042304  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4}
W20240503 02:17:08.051366  4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 50173us
I20240503 02:17:08.061005  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:08.091290  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:08.099889  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:08.121841  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:08.137900  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:08.153120  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:08.161633  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:08.176198  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:08.176687  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:08.245863  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:08.254338  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:08.267216  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:08.281724  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:08.284389  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:08.315584  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:08.327229  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:08.451319  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:08.460572  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:08.462568  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:08.516661  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:08.530089  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:08.532781  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:08.577495  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:08.588090  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:08.594306  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:08.595811  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:08.640259  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:08.694811  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:08.719921  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:08.813108  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:08.883201  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342): perf score=0.033336
W20240503 02:17:08.888296  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:08.990978  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(f0a088d230d0462980648628eae548d2): perf score=0.033371
I20240503 02:17:09.052809  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(51dbf0fd54b14c2fa21d3e62104c9b87): perf score=0.033397
I20240503 02:17:09.067560  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.184s	user 0.078s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15223,"dirs.run_cpu_time_us":322,"dirs.run_wall_time_us":34304,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1182,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":25218,"threads_started":2,"wal-append.queue_time_us":10278}
I20240503 02:17:09.072364  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342): 4838 bytes on disk
I20240503 02:17:09.075228  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":178,"lbm_reads_lt_1ms":4}
I20240503 02:17:09.090523  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(f0a088d230d0462980648628eae548d2): perf score=0.033393
W20240503 02:17:09.091837  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:09.155953  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:09.166249  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.175s	user 0.059s	sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6428,"dirs.run_cpu_time_us":331,"dirs.run_wall_time_us":10046,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3649,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13413,"threads_started":2,"wal-append.queue_time_us":6569}
I20240503 02:17:09.173003  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2): 4584 bytes on disk
I20240503 02:17:09.173977  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240503 02:17:09.206257  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:09.231756  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:09.259572  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:09.263125  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 171281us
I20240503 02:17:09.270521  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:09.276286  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:09.287305  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(51dbf0fd54b14c2fa21d3e62104c9b87) complete. Timing: real 0.234s	user 0.071s	sys 0.050s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15234,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":8796,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1059,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":103582,"threads_started":2,"wal-append.queue_time_us":76044}
I20240503 02:17:09.288880  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87): 4922 bytes on disk
I20240503 02:17:09.290280  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240503 02:17:09.298492  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:09.299654  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:09.307984  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:09.309715  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:09.311317  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:09.345762  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:09.361389  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:09.383615  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:09.394409  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:09.411979  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:09.418989  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:09.532155  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:17:09.541675  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.451s	user 0.070s	sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10878,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":3512,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3680,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":30240,"threads_started":2,"wal-append.queue_time_us":16027}
I20240503 02:17:09.543632  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2): 4584 bytes on disk
I20240503 02:17:09.544970  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240503 02:17:09.553452  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:17:09.737249  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:09.740514  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:09.780942  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:09.821053  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:09.875373  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:09.875447  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:09.931257  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:09.953240  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 115398us
I20240503 02:17:09.975986  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:10.043540  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:17:10.117740  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:10.155531  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:10.183202  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:10.193184  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:10.236534  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:10.243165  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:10.264490  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:10.296629  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:10.513120  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:10.538106  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:10.563091  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:10.582682  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:10.602135  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:10.620467  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:10.629588  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:10.656878  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:10.774320  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:17:10.918908  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:10.953438  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:10.960775  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:10.972712  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:10.988384  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:10.997730  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:11.006418  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:11.044982  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:11.047371  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:11.048069  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:11.080981  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 80203.6us
W20240503 02:17:11.088603  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:17:11.091468  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 52507.5us
W20240503 02:17:11.093149  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:17:11.118109  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:11.185264  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(0e61eb7d9e6041b6b99d5d5328943140): perf score=0.033345
I20240503 02:17:11.246702  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:11.263069  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:11.278374  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:11.284204  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:11.286203  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:11.304534  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(0e61eb7d9e6041b6b99d5d5328943140): perf score=0.033365
I20240503 02:17:11.307189  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(2de1d29dbea945eba58f8eadb019d766): perf score=0.033384
W20240503 02:17:11.319666  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:11.331660  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(0e61eb7d9e6041b6b99d5d5328943140) complete. Timing: real 0.146s	user 0.048s	sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3410,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":4794,"drs_written":1,"lbm_read_time_us":193,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64657,"threads_started":2,"wal-append.queue_time_us":38469}
I20240503 02:17:11.333495  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(2de1d29dbea945eba58f8eadb019d766): perf score=0.033380
I20240503 02:17:11.338001  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece): perf score=0.033380
W20240503 02:17:11.362432  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:11.418584  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:11.421387  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:11.435597  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:11.630133  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.323s	user 0.058s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11723,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":5417,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":49873,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":7936,"thread_start_us":32989,"threads_started":2,"wal-append.queue_time_us":19657}
I20240503 02:17:11.638969  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766): 4923 bytes on disk
I20240503 02:17:11.639712  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 02:17:11.640937  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece): perf score=0.033457
I20240503 02:17:11.655594  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(0e61eb7d9e6041b6b99d5d5328943140) complete. Timing: real 0.351s	user 0.083s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6843,"dirs.run_cpu_time_us":309,"dirs.run_wall_time_us":4847,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":23192,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14172,"threads_started":2,"wal-append.queue_time_us":7271}
I20240503 02:17:11.670521  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140): 4583 bytes on disk
I20240503 02:17:11.671366  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
I20240503 02:17:11.683919  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.346s	user 0.044s	sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4345,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1689,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5704,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41739,"threads_started":1,"wal-append.queue_time_us":41806}
I20240503 02:17:11.686200  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766): 4923 bytes on disk
I20240503 02:17:11.691162  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
I20240503 02:17:11.692453  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140): 4583 bytes on disk
I20240503 02:17:11.699324  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140) complete. Timing: real 0.006s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":192,"lbm_reads_lt_1ms":4}
W20240503 02:17:11.712844  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:11.745222  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.104s	user 0.043s	sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1658,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3712,"threads_started":1,"wal-append.queue_time_us":3905}
I20240503 02:17:11.747061  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece): 4838 bytes on disk
I20240503 02:17:11.748020  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":188,"lbm_reads_lt_1ms":4}
I20240503 02:17:11.748695  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.410s	user 0.054s	sys 0.038s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10536,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1583,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6053,"threads_started":2,"wal-append.queue_time_us":3439}
I20240503 02:17:11.755616  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece): 4838 bytes on disk
I20240503 02:17:11.756788  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":222,"lbm_reads_lt_1ms":4}
W20240503 02:17:11.823020  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:11.962219  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:12.002907  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:12.046494  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:17:12.074487  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:12.098898  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:12.132462  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:12.169376  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:12.170855  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:12.183640  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:12.196151  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 111151us
I20240503 02:17:12.201486  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:12.245106  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:12.263229  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:12.263870  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:12.270231  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:12.280624  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:12.292304  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:12.315992  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:12.368383  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:12.370723  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:12.560824  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:12.591714  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:12.628366  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:12.635970  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:12.641742  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:12.686296  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:17:12.687752  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:12.711194  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:12.726109  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:17:12.936226  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:12.966344  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:13.000638  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:13.009894  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:13.015197  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:13.039551  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:13.099650  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:13.102823  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:13.114873  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:13.228782  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:13.260357  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:13.284690  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:13.299881  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:13.314584  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:13.318357  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:13.345317  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:13.378561  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:13.387590  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:13.534437  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:13.540946  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:13.543154  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:13.561039  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:13.574599  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:13.583295  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:13.619613  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:13.626945  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:13.650882  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:13.660995  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:13.708385  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:13.742571  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:13.904193  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:13.948310  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 47625.9us
W20240503 02:17:13.948827  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 44648.9us
W20240503 02:17:13.962360  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:13.983888  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:13.985672  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:14.014166  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:14.047888  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:14.067775  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:14.103217  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:14.126220  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:14.167163  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:14.271517  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:14.320214  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:14.338454  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:14.348265  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:14.372169  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:14.385538  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:14.400010  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:14.401118  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:14.422933  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:14.483345  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:14.523808  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:17:14.544745  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:14.581943  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:14.601564  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:14.626075  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:14.652015  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:14.692373  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:14.699049  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:17:14.699829  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:14.757582  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:14.942523  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:15.036911  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:15.039297  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:15.072779  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:15.073983  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:15.093768  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:15.121166  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:15.159965  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:15.339484  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:15.362671  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:15.378585  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:15.400009  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:15.425740  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:15.461205  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:15.473217  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:15.518640  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:15.627907  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:15.640666  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:15.702651  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:15.721534  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:15.724861  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:15.729687  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:15.740092  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:15.764160  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:15.782389  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:15.837306  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:15.888242  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:15.986820  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:15.998217  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:16.017014  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:16.049389  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:16.089184  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:16.100180  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:16.155587  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:16.167479  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:16.178577  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:16.180483  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:16.183990  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:16.297784  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:16.358613  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:16.360095  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:16.390952  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:16.406076  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:16.420162  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:16.420950  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:16.431730  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:16.447944  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:16.459120  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:16.492094  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:16.504413  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:16.512248  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:16.553063  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:16.555853  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:16.602332  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:16.802708  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:16.818588  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:16.863777  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:17:16.879282  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:16.905458  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:16.906680  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:16.924679  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:16.944253  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:16.957240  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:16.968230  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:17.036492  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:17:17.073118  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:17:17.186141  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:17.210738  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:17.323308  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:17.344255  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:17.353155  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:17.369009  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:17.421530  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:17.422845  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:17.455175  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:17.500272  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:17.509017  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:17.527256  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:17.588409  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:17.602623  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:17.620641  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 80867.2us
I20240503 02:17:17.666357  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:17.671566  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:17.671819  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:17.775009  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:17.882071  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 02:17:18.001940  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:18.028184  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:18.032567  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:18.033293  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:18.095561  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:18.117317  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:18.136582  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:18.136723  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:18.161523  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:18.214234  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:18.214679  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.223057  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:18.227764  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.256155  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:18.269101  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.288178  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:18.294091  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.319471  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:18.321547  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:18.510943  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.558725  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:18.582774  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:18.583667  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:18.596138  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:18.610395  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:18.615325  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:18.638854  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.652886  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:18.697149  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:18.706663  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:18.913507  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:19.050737  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:19.072072  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:19.083580  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:19.083711  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:19.107244  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:19.178182  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:19.190637  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:19.215790  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:19.217538  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:19.221159  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:19.221649  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:19.221850  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:19.229626  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:19.251637  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:19.279726  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:19.282951  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:19.360505  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:17:19.447853  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:19.492890  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:17:19.616858  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:17:19.639806  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:19.662819  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:19.671146  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:19.695741  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:19.710325  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:19.710958  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:19.711398  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:19.746088  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:19.852965  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:19.875744  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:19.915910  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:19.931797  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:19.941991  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:19.944350  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:19.961869  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:19.983037  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:19.994189  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 55094.4us
I20240503 02:17:20.172384  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:20.175158  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:20.181170  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:20.203471  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:20.209739  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:20.250310  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:20.251209  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:20.279726  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:20.540244  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:17:20.574787  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:20.631058  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:20.659394  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:20.681277  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:20.703087  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:20.711285  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:20.741250  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:20.745466  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:20.761286  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:20.763469  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:17:20.839699  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 75469.3us
W20240503 02:17:20.868397  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:20.900918  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:20.912936  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:20.920959  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:20.943092  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:20.971906  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:20.981683  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:21.002494  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:21.023041  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:17:21.130206  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:21.200433  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:21.200834  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:21.220633  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:21.221028  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:21.223434  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:21.241842  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:21.268591  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:21.302798  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:17:21.348265  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:17:21.400708  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:21.448851  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:21.479099  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:21.485492  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:21.534319  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:21.557406  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:21.581961  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:21.622474  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:21.833242  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:21.840818  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(190ebb1335d64b279a41fa38feff22b4): perf score=0.033363
W20240503 02:17:21.846091  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:21.885658  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:22.039031  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:22.034189  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.192s	user 0.075s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":3460,"dirs.queue_time_us":3552,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1032,"drs_written":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4,"lbm_write_time_us":812,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13903,"threads_started":3,"wal-append.queue_time_us":6768}
I20240503 02:17:22.045667  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:22.046944  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4): 4838 bytes on disk
I20240503 02:17:22.049844  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(190ebb1335d64b279a41fa38feff22b4) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 02:17:22.075397  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:22.085916  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:22.109864  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:22.166859  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:22.214095  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:22.246083  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:22.285964  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:17:22.367125  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:22.383424  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:22.409989  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:22.439682  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 02:17:22.440798  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:22.481016  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:22.496995  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:22.548396  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:22.567183  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:22.660264  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(1eeb69e5eb3c459ca749aa37672ec057): perf score=0.033358
W20240503 02:17:22.686281  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:22.746279  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:22.775508  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:22.804330  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(0137e646f1bd452b950d870107afa060): perf score=0.033400
I20240503 02:17:22.844367  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(1eeb69e5eb3c459ca749aa37672ec057) complete. Timing: real 0.183s	user 0.076s	sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15629,"dirs.queue_time_us":45130,"dirs.run_cpu_time_us":305,"dirs.run_wall_time_us":1039,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2332,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":69694,"threads_started":3,"wal-append.queue_time_us":6580}
I20240503 02:17:22.846892  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(f084f1783d4d477e9c3791984e681f4c): perf score=0.033387
I20240503 02:17:22.852876  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:22.866643  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:22.874397  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:22.887455  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:22.912159  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:22.926997  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:22.928828  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:23.043028  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:23.116891  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.130877  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.277s	user 0.086s	sys 0.047s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":2250,"drs_written":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1129,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60837,"threads_started":1,"wal-append.queue_time_us":60957}
I20240503 02:17:23.133136  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057): 4922 bytes on disk
I20240503 02:17:23.134513  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240503 02:17:23.135977  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c): 4585 bytes on disk
I20240503 02:17:23.136797  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
W20240503 02:17:23.163131  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.173821  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:23.262205  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:23.282567  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:23.289352  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:23.337810  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:23.361614  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:23.440300  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.636s	user 0.041s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4535,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":2441,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7280,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":24527,"threads_started":2,"wal-append.queue_time_us":18232}
I20240503 02:17:23.441951  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060): 4923 bytes on disk
I20240503 02:17:23.451443  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:23.455581  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 02:17:23.457053  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(8870feaf47a64a929dd2e584ece47218): perf score=0.033422
W20240503 02:17:23.478787  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.482904  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:23.566717  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(62a3dcc2de594c8982477a0229f71191): perf score=0.033370
W20240503 02:17:23.586803  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.640431  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(586ebf467f0f427fa62dee6223a39848): perf score=0.033355
I20240503 02:17:23.647845  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.188s	user 0.053s	sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1840,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1071,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":42375,"threads_started":1,"wal-append.queue_time_us":42279}
I20240503 02:17:23.649380  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218): 4923 bytes on disk
I20240503 02:17:23.666021  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20240503 02:17:23.670804  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:23.672885  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.802582  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:23.809906  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:23.819998  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.820179  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:23.832260  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.265s	user 0.063s	sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":4224,"dirs.queue_time_us":6569,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":7119,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1256,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97747,"threads_started":3,"wal-append.queue_time_us":74700}
I20240503 02:17:23.834599  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(51224b6355ed41dbb0703b7feb73c223): perf score=0.033397
I20240503 02:17:23.842321  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:23.861457  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:23.870568  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:23.882340  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:23.919485  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:24.031515  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.391s	user 0.065s	sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5687,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":2652,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":996,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12534,"threads_started":2,"wal-append.queue_time_us":6561}
I20240503 02:17:24.033129  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848): 4923 bytes on disk
I20240503 02:17:24.033862  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.035220  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(0137e646f1bd452b950d870107afa060): perf score=0.033460
W20240503 02:17:24.053532  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:24.182593  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:24.199491  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:24.242712  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:24.262159  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:24.285271  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:24.300236  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:24.332741  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.497s	user 0.074s	sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1862,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":26037,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26808,"threads_started":1,"wal-append.queue_time_us":23076}
I20240503 02:17:24.334282  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(f0a088d230d0462980648628eae548d2): perf score=0.033416
I20240503 02:17:24.361389  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:24.363651  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:24.388228  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.349s	user 0.039s	sys 0.031s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":732,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1729,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1090,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":61511,"threads_started":1,"wal-append.queue_time_us":41836}
I20240503 02:17:24.389918  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060): 4923 bytes on disk
I20240503 02:17:24.394418  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(0137e646f1bd452b950d870107afa060) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.445487  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.107s	user 0.056s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6853,"threads_started":1,"wal-append.queue_time_us":6716}
I20240503 02:17:24.447144  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1): perf score=0.033445
W20240503 02:17:24.449960  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 43948.7us
W20240503 02:17:24.553439  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:17:24.558274  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:17:24.579844  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.126s	user 0.059s	sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3519,"threads_started":1,"wal-append.queue_time_us":3393}
I20240503 02:17:24.582414  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191): 4923 bytes on disk
I20240503 02:17:24.585196  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.586666  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223): 4838 bytes on disk
I20240503 02:17:24.587229  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.590092  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2): 4584 bytes on disk
I20240503 02:17:24.590680  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(f0a088d230d0462980648628eae548d2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.593055  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1): 4583 bytes on disk
I20240503 02:17:24.593720  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(a1949a5c7cc741b19da021eccb7d0dd1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 02:17:24.630044  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 02:17:24.646883  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342): perf score=0.033368
I20240503 02:17:24.675915  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab): perf score=0.033392
I20240503 02:17:24.792886  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.117s	user 0.036s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":115796,"compiler_manager_pool.run_cpu_time_us":53,"compiler_manager_pool.run_wall_time_us":53,"dirs.queue_time_us":4185,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":26798,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":970,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14045,"threads_started":2,"wal-append.queue_time_us":7018}
I20240503 02:17:24.804683  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.157s	user 0.045s	sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":107984,"compiler_manager_pool.run_cpu_time_us":64,"compiler_manager_pool.run_wall_time_us":65,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":900,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5897,"threads_started":1,"wal-append.queue_time_us":10853}
I20240503 02:17:24.808560  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(02e0b954008c4f30af414da6cb5d59c2): perf score=0.033369
I20240503 02:17:24.813081  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342): 4838 bytes on disk
I20240503 02:17:24.813777  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 02:17:24.817306  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:24.896296  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:24.907424  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:24.923411  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:24.932022  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(02e0b954008c4f30af414da6cb5d59c2) complete. Timing: real 0.123s	user 0.048s	sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1693,"drs_written":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16883,"threads_started":1,"wal-append.queue_time_us":16031}
I20240503 02:17:24.935463  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab): 4923 bytes on disk
I20240503 02:17:24.936291  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(a6e0d39087944e4ea13a17ca4af478ab) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.938243  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2): 4838 bytes on disk
I20240503 02:17:24.938932  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 02:17:24.966907  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:24.992940  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:25.004467  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:25.022737  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:25.045246  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:25.098603  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(734027e195414b15811ac9d32b132db8): perf score=0.033358
W20240503 02:17:25.149847  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:25.192414  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(75ba33132e784f408f3e26882aea3d15): perf score=0.033385
W20240503 02:17:25.232818  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:25.279839  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.181s	user 0.054s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9071,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":11782,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35187,"threads_started":2,"wal-append.queue_time_us":23636}
I20240503 02:17:25.286423  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8): 4583 bytes on disk
I20240503 02:17:25.287160  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
I20240503 02:17:25.289299  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.097s	user 0.067s	sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1668,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":979,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9959,"threads_started":1,"wal-append.queue_time_us":9832}
I20240503 02:17:25.291268  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15): 4923 bytes on disk
I20240503 02:17:25.294772  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":473,"lbm_reads_lt_1ms":4}
W20240503 02:17:25.298671  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:25.412478  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:25.426501  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:25.445117  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:25.482079  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:25.511296  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:25.570004  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:25.577600  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:25.625808  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:25.692380  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:25.725432  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:25.735138  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:25.747426  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:25.749176  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:25.771003  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:25.796267  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:25.805562  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:25.843955  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:17:25.858973  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 83825.1us
I20240503 02:17:25.868448  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:26.079149  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:17:26.105077  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:26.109613  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:26.115360  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:26.116614  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:26.122236  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:26.126981  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:26.158681  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:26.196468  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:26.196902  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:26.273515  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:17:26.368570  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:26.370695  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:26.435098  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:26.450816  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:26.468840  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:26.473269  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:26.539598  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 49269.4us
I20240503 02:17:26.545256  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:26.546476  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:26.563351  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:26.575640  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(51dbf0fd54b14c2fa21d3e62104c9b87): perf score=0.033376
I20240503 02:17:26.724552  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(51dbf0fd54b14c2fa21d3e62104c9b87) complete. Timing: real 0.149s	user 0.039s	sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5092,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":13172,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":46208,"threads_started":2,"wal-append.queue_time_us":23645}
I20240503 02:17:26.728030  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:26.728128  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(9b822c691d824b2ba25e967535c9734e): perf score=0.033348
I20240503 02:17:26.754978  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:26.856333  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:26.861122  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:26.872013  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240503 02:17:26.884497  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.156s	user 0.069s	sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1986,"drs_written":1,"lbm_read_time_us":2083,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1195,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":10358,"threads_started":1,"wal-append.queue_time_us":10240}
I20240503 02:17:26.886698  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87): 4922 bytes on disk
I20240503 02:17:26.887887  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 02:17:26.889289  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(75ba33132e784f408f3e26882aea3d15): perf score=0.033379
W20240503 02:17:26.906456  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:26.911263  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:26.947712  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:26.958133  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:27.097393  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.204s	user 0.046s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8764,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":3418,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6388,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5150,"threads_started":1,"wal-append.queue_time_us":2956}
I20240503 02:17:27.099565  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15): 4923 bytes on disk
I20240503 02:17:27.100709  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(75ba33132e784f408f3e26882aea3d15) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":183,"lbm_reads_lt_1ms":4}
I20240503 02:17:27.102674  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e): 4583 bytes on disk
I20240503 02:17:27.104151  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":594,"lbm_reads_lt_1ms":4}
I20240503 02:17:27.204797  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:27.222272  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:27.233777  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:27.235992  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:27.243309  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:27.260375  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:17:27.285008  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 65694.1us
W20240503 02:17:27.287051  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 112586us
W20240503 02:17:27.302731  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 130496us
W20240503 02:17:27.308715  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:17:27.350772  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:27.355345  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:27.487954  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:27.553174  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(771a4190142b4fd5a9479417122c0881): perf score=0.033395
I20240503 02:17:27.627175  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:27.662658  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:27.662858  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:27.663166  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:27.665846  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:27.721174  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:27.737970  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:27.752482  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:27.797449  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.244s	user 0.035s	sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61731,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":7893,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7650,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":37918,"threads_started":2,"wal-append.queue_time_us":28358}
I20240503 02:17:27.802841  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881): 4581 bytes on disk
I20240503 02:17:27.805389  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.000s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240503 02:17:27.806170  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:27.811293  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0): perf score=0.033386
W20240503 02:17:27.819285  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 66816.3us
I20240503 02:17:27.844183  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:27.861692  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:27.901777  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:27.904729  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:27.918874  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:17:27.943537  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:27.968711  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:27.978047  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:27.991014  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:28.056107  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.244s	user 0.037s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":29062,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":4004,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":66851,"threads_started":2,"wal-append.queue_time_us":34493}
I20240503 02:17:28.057756  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(9253de3f798f4a208d566765ed2ba54d): perf score=0.033450
W20240503 02:17:28.069507  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:28.070812  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:28.115713  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.057s	user 0.033s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":5031,"drs_written":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7275,"threads_started":1,"wal-append.queue_time_us":7366}
I20240503 02:17:28.117635  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0): 4838 bytes on disk
I20240503 02:17:28.118714  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4}
I20240503 02:17:28.120362  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d): 4836 bytes on disk
I20240503 02:17:28.121163  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(9253de3f798f4a208d566765ed2ba54d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240503 02:17:28.224822  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:28.247363  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:28.254925  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:28.266076  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:28.290385  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:28.303983  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:28.337828  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 50189.5us
I20240503 02:17:28.365796  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:28.369484  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:17:28.370366  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:17:28.485134  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:28.507545  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:28.529868  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:28.547638  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:28.558657  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:28.575583  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:28.596849  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:28.611166  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:28.641784  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 69017us
W20240503 02:17:28.660257  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:28.663164  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:28.721752  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:17:28.725355  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:28.809407  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:28.864984  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:28.880640  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:28.927114  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:28.930574  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:28.942971  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:28.981629  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:29.025704  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:29.123365  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(2de1d29dbea945eba58f8eadb019d766): perf score=0.033400
W20240503 02:17:29.156320  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:29.191778  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:29.215291  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:29.244041  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:29.253180  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:29.269590  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:29.319064  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 51903.8us
I20240503 02:17:29.327963  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:29.329135  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.205s	user 0.061s	sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":403,"dirs.run_cpu_time_us":327,"dirs.run_wall_time_us":2065,"drs_written":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":60434,"threads_started":2,"wal-append.queue_time_us":59701}
I20240503 02:17:29.330612  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766): 4923 bytes on disk
I20240503 02:17:29.331240  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(2de1d29dbea945eba58f8eadb019d766) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:17:29.398922  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:29.406088  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:29.567505  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(43e248f0cb764d229802851c4e73cc96): perf score=0.033346
I20240503 02:17:29.586045  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:29.597167  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:29.662267  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.094s	user 0.063s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6516,"dirs.run_cpu_time_us":360,"dirs.run_wall_time_us":14019,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1920,"lbm_writes_lt_1ms":23,"mutex_wait_us":17,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13524,"threads_started":2,"wal-append.queue_time_us":6650}
I20240503 02:17:29.663949  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96): 4838 bytes on disk
I20240503 02:17:29.664860  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
W20240503 02:17:29.666440  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:17:29.718952  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:29.726265  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:29.741101  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:29.760752  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:29.805991  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:29.816246  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:29.816315  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:29.884640  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(f084f1783d4d477e9c3791984e681f4c): perf score=0.033360
I20240503 02:17:29.917624  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6): perf score=0.033391
I20240503 02:17:30.022703  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:30.034102  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.149s	user 0.045s	sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7327,"dirs.run_cpu_time_us":356,"dirs.run_wall_time_us":8344,"drs_written":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4,"lbm_write_time_us":896,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26662,"threads_started":2,"wal-append.queue_time_us":6090}
I20240503 02:17:30.035920  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c): 4585 bytes on disk
I20240503 02:17:30.038247  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(f084f1783d4d477e9c3791984e681f4c) complete. Timing: real 0.001s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:17:30.053822  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:30.055075  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:30.079447  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:30.084312  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:30.095139  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.177s	user 0.076s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1689,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1145,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":36227,"threads_started":1,"wal-append.queue_time_us":35022}
I20240503 02:17:30.096688  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6): 4838 bytes on disk
I20240503 02:17:30.100023  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 02:17:30.115731  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:30.149971  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:30.205901  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:30.326435  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:30.353196  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(9b822c691d824b2ba25e967535c9734e): perf score=0.033350
I20240503 02:17:30.395097  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:30.406498  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:30.417050  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:30.427577  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:30.429775  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:30.459811  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:30.481443  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:30.524302  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:30.614825  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 91778.5us
W20240503 02:17:30.670722  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:30.690065  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.336s	user 0.065s	sys 0.048s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":851,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":3227,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2106,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":199237,"threads_started":1,"wal-append.queue_time_us":196530}
I20240503 02:17:30.693758  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e): 4583 bytes on disk
I20240503 02:17:30.699883  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(9b822c691d824b2ba25e967535c9734e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":503,"lbm_reads_lt_1ms":4}
I20240503 02:17:30.702481  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2): perf score=0.033391
W20240503 02:17:30.726263  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:30.761762  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:30.765513  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:30.776804  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:30.779963  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:30.790215  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:30.793702  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:30.804136  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:30.830085  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:30.840719  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(43e248f0cb764d229802851c4e73cc96): perf score=0.033393
I20240503 02:17:30.846387  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:30.908196  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:30.925257  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.013622  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.311s	user 0.034s	sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5128,"dirs.run_cpu_time_us":271,"dirs.run_wall_time_us":5798,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1082,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29196,"threads_started":2,"wal-append.queue_time_us":7837}
I20240503 02:17:31.016404  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2): 4583 bytes on disk
I20240503 02:17:31.017164  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(913d0aa014ad4bcaac905b5fdaa296d2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 02:17:31.046094  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.205s	user 0.040s	sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4087,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":3514,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1033,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17703,"threads_started":2,"wal-append.queue_time_us":10731}
I20240503 02:17:31.049229  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96): 4838 bytes on disk
W20240503 02:17:31.051846  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.053983  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(43e248f0cb764d229802851c4e73cc96) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240503 02:17:31.064685  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.131763  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:31.141186  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:31.155092  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.157234  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:31.178825  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:31.192350  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:31.204897  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:31.239521  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:31.307125  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(592cf4d5b35e49bfa4173343d64367f4): perf score=0.033400
I20240503 02:17:31.392180  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(592cf4d5b35e49bfa4173343d64367f4) complete. Timing: real 0.085s	user 0.047s	sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1695,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1083,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19963,"threads_started":1,"wal-append.queue_time_us":20187}
I20240503 02:17:31.393668  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4): 4583 bytes on disk
I20240503 02:17:31.394434  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20240503 02:17:31.494971  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.515790  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:31.524776  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:31.528539  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:31.555395  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:31.586422  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:31.607468  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:31.615738  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:31.654812  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:31.666648  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:31.889652  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:31.921909  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:31.922326  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:31.942056  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:31.960759  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:31.992535  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:32.022472  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:32.027451  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702): perf score=0.033351
W20240503 02:17:32.088992  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:32.143620  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.116s	user 0.039s	sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11678,"dirs.run_cpu_time_us":292,"dirs.run_wall_time_us":12271,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1155,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":447744,"thread_start_us":16307,"threads_started":2,"wal-append.queue_time_us":6546}
I20240503 02:17:32.145176  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702): 4838 bytes on disk
I20240503 02:17:32.146059  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":148,"lbm_reads_lt_1ms":4}
I20240503 02:17:32.297139  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:32.304311  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:32.339015  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:32.376241  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:17:32.384228  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:32.424171  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:32.501024  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:32.522068  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:17:32.553099  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702): perf score=0.033351
I20240503 02:17:32.567709  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:32.619064  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:32.700397  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:32.707751  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.154s	user 0.045s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":39500,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":8169,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67429,"threads_started":2,"wal-append.queue_time_us":27501}
I20240503 02:17:32.723923  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702): 4838 bytes on disk
I20240503 02:17:32.725534  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(7a62b3ef44e94e92b0feed9c228f5702) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 02:17:32.726260  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:32.740756  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:32.754230  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:32.773852  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:32.821491  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:32.827385  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:17:32.830987  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:32.928251  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:32.928364  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:32.963858  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:32.981031  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:32.981528  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:33.005913  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:33.037086  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:33.078099  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:33.079063  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:33.156502  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:33.158138  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:33.184675  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:33.288210  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:33.288708  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:17:33.316430  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:33.349587  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:33.358937  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:33.361666  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:33.375614  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:33.417078  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:33.451589  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:33.469175  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:17:33.481350  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:33.499831  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:33.554208  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:33.586413  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:17:33.737653  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:33.744927  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:33.767323  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:33.806289  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:33.831457  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:33.847285  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:33.894407  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:33.942529  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:33.948027  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:33.954047  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:33.966482  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:33.975548  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:33.990443  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:33.994858  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:34.046489  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:17:34.080054  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:34.341432  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:34.367132  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:34.377667  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:34.381443  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:34.385825  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:34.408706  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:34.415966  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:34.438275  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:34.497896  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:34.608618  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:34.609133  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:34.609848  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:34.610929  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:34.612180  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:34.616035  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:34.619768  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:34.755832  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:17:34.922398  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:34.926033  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:34.940780  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:34.981961  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:34.991551  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:34.998656  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:35.007922  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:35.023428  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:35.045862  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:17:35.220813  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:17:35.247596  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:35.274405  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:35.279476  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:35.281641  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:35.311937  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:35.324807  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:35.361454  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:35.467044  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:35.480546  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:35.520412  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:35.545179  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:35.553920  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:35.560484  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:35.576442  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:35.626168  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:35.626665  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:35.653148  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:35.670953  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:35.709508  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:35.783217  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:35.840401  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:35.850255  4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 44815.5us
I20240503 02:17:35.929428  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:35.935307  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:35.971483  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:35.995390  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:36.008539  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:36.025235  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:36.032816  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:36.034642  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:17:36.037003  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:36.090912  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:17:36.220727  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:36.222779  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:17:36.248045  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:36.256827  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:36.275218  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:36.303153  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:36.343283  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:36.372737  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:36.393529  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:36.406585  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:36.426216  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:36.452932  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:36.470835  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:36.501854  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:36.513475  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:36.514302  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:36.523824  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:36.797768  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:36.800491  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:36.822680  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:36.825433  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:36.827121  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:36.847929  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:36.884594  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 132437us
I20240503 02:17:36.885999  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:36.975253  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 46944.5us
W20240503 02:17:37.102151  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:37.214738  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:37.230403  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:17:37.235447  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:37.260859  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:37.262785  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:37.279397  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:37.313696  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 53893.1us
I20240503 02:17:37.333122  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:37.355816  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:37.403290  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:17:37.531780  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:37.543113  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:37.556224  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:37.574805  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:37.624331  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:17:37.633549  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:37.646723  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:37.668900  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:37.695148  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:37.787751  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:38.017745  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:38.018594  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:38.020815  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:38.042006  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:38.072957  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:38.134306  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:38.138088  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:38.174582  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:38.189951  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:38.191246  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:17:38.197458  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:38.205215  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:17:38.210692  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:17:38.248231  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:17:38.274853  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:38.358882  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:38.374094  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:38.378769  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:38.420349  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:38.430892  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:38.445766  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:38.506004  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:38.516512  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:38.517484  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:38.544059  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:38.596071  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:38.688489  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:38.708179  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:38.765172  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:17:38.769656  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:38.783294  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:38.898583  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:38.995160  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:39.004941  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:39.061161  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:39.194845  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:39.210453  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:39.214870  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:39.230662  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:39.242830  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:39.255335  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:39.308606  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:39.309499  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:39.493305  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:39.654580  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:39.687597  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:39.693687  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:39.705994  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:39.706873  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:39.716774  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:39.729807  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:39.744364  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:39.748211  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:39.770946  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:39.783972  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:39.806376  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:39.816102  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:39.859212  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:40.111688  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:40.155455  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:17:40.274582  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:40.281767  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:40.288152  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:40.295302  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:40.308282  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:40.328483  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:40.386178  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:40.388358  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:40.398375  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:40.541622  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:40.546128  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:40.557747  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:40.594393  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:40.601022  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:40.623891  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:40.677929  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:40.678171  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:17:40.724298  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:17:40.724678  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:40.741444  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:40.746968  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:40.758131  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:17:40.872598  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:40.873811  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:40.975713  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:40.986591  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:41.010341  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:41.012441  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:41.017621  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:41.018848  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:41.030562  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:41.229715  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:41.229751  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:41.246045  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:17:41.255517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:41.260485  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:41.287755  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:41.289032  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:41.300635  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:41.329723  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:41.467304  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:41.606029  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:41.612207  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:41.628796  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:41.633149  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:41.671422  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:41.686195  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:41.699225  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:41.731945  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:17:41.811880  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:41.925856  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:41.957309  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:41.999166  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:42.014676  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:42.022946  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:42.063522  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:42.120097  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:42.122036  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:42.299069  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:42.305439  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:42.318418  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:42.340901  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:42.342331  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:42.360800  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:42.380229  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:42.380790  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:42.495821  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:42.559855  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:42.561584  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:42.622390  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:42.636162  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:42.675117  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:42.686805  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:17:42.689453  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:42.856057  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:17:42.912050  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:42.960477  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:17:42.962282  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:42.967753  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:43.004673  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:43.013474  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:43.014678  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:43.016840  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:43.043886  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:43.060129  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:43.111982  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:43.152549  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:17:43.235553  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:43.236882  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:17:43.244328  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:43.246840  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:43.291380  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:43.298393  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:43.299955  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:43.323603  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:43.357091  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:43.386518  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:43.428151  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:43.432085  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:43.590710  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:43.609366  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:43.634820  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:43.643527  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:43.661288  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 57080.3us
W20240503 02:17:43.668339  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:43.683214  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:43.687912  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:43.712965  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:43.735524  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:43.901532  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:44.021304  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:44.050240  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:44.057814  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:44.059072  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:44.114799  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:44.127743  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:44.169857  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:44.219416  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:17:44.234709  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:44.338467  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:44.346267  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:44.394647  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:44.455248  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:44.472517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:44.503764  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:17:44.536495  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:44.544636  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:44.554085  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:44.665355  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:44.684885  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:44.690930  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:44.751541  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:44.773183  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:44.827076  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:44.867571  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:44.962723  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:45.057950  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:45.086833  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:45.087973  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:45.105631  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:45.124580  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:45.163158  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:45.177707  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:45.287178  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:45.319876  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:45.329538  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:45.340592  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:45.345961  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:45.389408  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:45.398592  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:45.408234  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:45.416252  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:45.437906  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:45.506415  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:45.600680  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:17:45.625655  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:17:45.646221  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:17:45.663722  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:45.686954  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:45.711463  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:45.746683  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:45.753705  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:45.760720  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:45.762182  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:45.778667  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:45.789423  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:45.829378  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:45.840329  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 113358us
W20240503 02:17:46.020967  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:17:46.035494  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:46.042703  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:46.095950  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:46.109712  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:46.127969  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:46.182355  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:46.221485  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:46.295675  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:46.298835  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:46.338639  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:46.338811  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:46.372562  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:46.377897  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:46.433362  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:46.448879  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:46.468073  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:46.505560  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:46.578384  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:46.661433  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:46.673902  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:46.683173  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:46.686787  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:46.692854  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:46.693719  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:46.725548  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:46.774763  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:46.872910  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:47.064859  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:47.071091  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:47.079855  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:47.082289  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:47.093228  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:47.114609  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:47.153170  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:47.158373  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:47.318574  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:47.331732  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:47.338217  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:47.338640  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:47.375650  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:47.377059  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:47.398176  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:47.440768  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:47.616626  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:47.655895  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:47.676326  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:47.682540  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:47.695181  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:17:47.709354  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:47.727155  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:47.746297  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:47.749244  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:47.787384  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:17:47.807981  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:47.861949  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:17:47.960250  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:48.092089  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:48.138003  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:48.170400  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:48.172103  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:48.193536  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:17:48.219135  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:48.221163  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:48.248796  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:17:48.280508  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:48.320165  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:17:48.327023  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:48.379910  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:48.380987  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:48.504514  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:48.549373  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:48.562390  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:48.572896  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:48.573086  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:48.589691  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:48.654045  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:48.666468  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:48.689803  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:48.690042  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:48.726217  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:48.751638  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:48.790401  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:48.823763  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:48.831363  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:48.839277  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:17:48.879752  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:48.887282  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:49.159608  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:49.180392  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:49.187815  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:49.231200  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:49.244298  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:49.261147  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:49.273423  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:49.291510  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:49.400432  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:49.424655  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:17:49.642895  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:49.701982  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:49.736523  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:49.761940  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:49.778623  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:49.859213  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:49.860456  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:49.892163  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:49.896239  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:49.897778  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:49.939936  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:49.985638  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:50.012969  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:50.063560  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:50.064899  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:50.262321  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:50.271427  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:50.283761  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:50.289275  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:50.299152  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:50.327895  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:50.343542  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 56679.4us
I20240503 02:17:50.348747  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:50.362084  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:50.384771  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:50.403450  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:50.477236  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:50.508205  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:50.532824  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:17:50.597942  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:50.598042  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:50.614622  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:17:50.648617  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:50.655189  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:50.697559  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:50.715816  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:50.740516  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:50.749966  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:50.847470  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:50.882578  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:17:50.911247  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 83753.5us
W20240503 02:17:50.967414  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 51931.1us
I20240503 02:17:51.094885  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:51.104583  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:51.140334  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:51.146132  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:17:51.147991  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:51.150930  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:51.153970  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:17:51.173581  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:51.190879  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:51.206858  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:17:51.208278  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:51.274926  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:51.354466  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:51.363435  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:51.376047  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:51.384681  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:51.400661  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:51.426209  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:51.466881  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:51.620651  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:17:51.706189  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:51.712644  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:51.751003  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6): perf score=0.033379
I20240503 02:17:51.752177  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:51.769093  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:51.787822  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:51.806779  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:51.863991  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 136517us
W20240503 02:17:51.889856  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:17:51.891155  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:51.904116  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 66456.4us
W20240503 02:17:51.911381  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:52.062178  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.311s	user 0.064s	sys 0.053s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16996,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":4276,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":83770,"threads_started":2,"wal-append.queue_time_us":63710}
I20240503 02:17:52.063683  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6): 4838 bytes on disk
I20240503 02:17:52.064428  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(ae239d73f0bc4d9aac025e40489e81a6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240503 02:17:52.204674  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:52.210621  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:52.223773  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:52.265677  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:52.283528  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:52.297021  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 107714us
W20240503 02:17:52.298830  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:52.339946  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:52.358839  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:52.556329  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:52.567065  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:52.592790  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:52.604054  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:52.611349  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:52.639802  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:52.668879  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:52.693710  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:52.714828  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:52.730404  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:52.760192  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:17:52.785549  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 104260us
I20240503 02:17:52.790299  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:52.837059  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:52.854177  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:17:52.862468  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:52.890151  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:52.893297  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:52.928226  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:52.972994  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:53.101718  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:53.101867  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:53.130991  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:53.134928  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:53.136107  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:53.152920  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:53.163312  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:53.163561  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:53.190989  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:53.198274  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:17:53.252326  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:17:53.376484  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:17:53.392624  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:53.394129  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:53.483256  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:53.495308  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:53.496129  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:53.517836  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:53.527592  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:53.563686  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:53.582619  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:53.583201  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:53.610574  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:53.649111  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:53.659405  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:53.678889  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:53.701633  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:53.703347  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:53.726091  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:53.756831  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:53.767733  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:53.841784  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:17:54.091764  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:54.106950  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:54.122355  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:54.122684  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:54.123736  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:54.165606  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:54.170274  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:54.171363  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:54.304672  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:54.348915  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:54.383949  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:54.404029  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:54.410351  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:54.427013  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:54.464990  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:54.476559  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:17:54.507220  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:54.581862  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling FlushMRSOp(62a3dcc2de594c8982477a0229f71191): perf score=0.033392
W20240503 02:17:54.642856  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:54.692701  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: FlushMRSOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.110s	user 0.046s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":821,"dirs.run_cpu_time_us":338,"dirs.run_wall_time_us":12339,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1005,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3881,"threads_started":2,"wal-append.queue_time_us":3239}
I20240503 02:17:54.694258  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191): 4923 bytes on disk
I20240503 02:17:54.695125  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":164,"lbm_reads_lt_1ms":4}
I20240503 02:17:54.791961  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:54.826004  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:54.919178  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:54.967077  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:55.019750  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:55.028568  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:55.036772  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(d2351807a2ed47419f8d674fbeb9d0ec): perf score=0.033360
I20240503 02:17:55.041188  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:55.077378  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:17:55.128873  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:55.162076  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(d2351807a2ed47419f8d674fbeb9d0ec) complete. Timing: real 0.125s	user 0.043s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":403,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":2797,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":525,"threads_started":2,"wal-append.queue_time_us":377}
I20240503 02:17:55.164052  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec): 4583 bytes on disk
I20240503 02:17:55.164780  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 02:17:55.184235  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:55.226065  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:55.230657  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:55.263342  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:55.283640  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:55.321008  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 60712us
I20240503 02:17:55.342968  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:55.347522  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:55.400594  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:17:55.445500  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:55.487335  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:55.518819  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:55.552310  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:55.563227  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:55.598832  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:55.610462  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:55.626159  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:55.642530  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:55.660634  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:55.672703  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:17:55.674288  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 59293.6us
W20240503 02:17:55.679101  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:17:55.784076  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:17:55.945088  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:56.003525  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:56.010957  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:17:56.027983  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:56.057289  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:56.079464  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:56.084209  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:56.104089  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:56.131222  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:56.176260  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 51931.3us
W20240503 02:17:56.196235  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:56.246795  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:17:56.297722  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:56.352422  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:56.363514  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:56.369045  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:56.391520  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:56.415663  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:56.463404  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:56.504285  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:17:56.516486  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:56.531737  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:56.582021  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:56.586325  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:56.601644  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:56.625571  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:56.646806  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:56.679033  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:56.814644  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:56.960637  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:56.998823  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:57.000357  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:17:57.020702  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:57.020833  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:57.031798  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:57.045651  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:57.059693  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:57.072587  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:57.091152  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:57.355998  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:57.386967  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:57.411111  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:57.464702  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:57.516571  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:57.519259  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:57.578696  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:57.581547  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:57.623426  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:57.631155  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:57.661863  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:57.666218  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:57.699451  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:57.702849  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:57.731062  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 62228.2us
I20240503 02:17:57.735831  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:57.770586  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:57.814253  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:17:57.914237  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 61314.5us
W20240503 02:17:57.936157  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:57.942672  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:17:57.953866  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:17:57.999161  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:58.014684  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:58.097862  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:58.097848  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:58.105175  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:17:58.126734  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:58.151901  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:58.162628  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:58.175004  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:58.201417  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:58.430213  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:58.432804  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:58.463120  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:58.465381  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:58.468946  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:58.474210  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:58.486599  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:58.490187  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:58.497862  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:17:58.522079  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:58.526829  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:58.545630  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:58.768499  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:17:58.821424  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:58.850656  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:58.915333  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:58.932396  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:58.969779  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:17:59.017554  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:17:59.038332  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:59.045425  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:59.046420  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:59.084721  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:59.099296  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:17:59.171746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:59.175128  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:59.179737  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:59.223212  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:17:59.230710  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:17:59.434455  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:17:59.503401  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:17:59.515373  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:17:59.541393  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:17:59.542001  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:17:59.558553  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:17:59.563292  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:17:59.578915  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:17:59.585659  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:17:59.592275  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:17:59.662751  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:17:59.947429  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:00.090401  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:00.120005  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:00.124011  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:00.133986  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:00.135064  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:00.154168  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:00.158627  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:00.202011  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:00.220314  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:00.222209  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:00.275960  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:00.298316  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:00.308429  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:00.338980  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:00.340330  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:00.357224  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 161412us
I20240503 02:18:00.366428  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:00.371150  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:00.421257  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:00.422786  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:00.423069  2391 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:00.423259  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:00.428710  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:00.433735  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:00.448673  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 55068.4us
W20240503 02:18:00.530092  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:00.588783  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:18:00.640815  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:18:00.808084  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:00.813359  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:18:00.858187  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:00.888974  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:18:00.907416  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:01.162393  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:01.198612  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:01.199828  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:01.212662  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:01.217368  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:01.219044  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:01.237258  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:01.286473  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:18:01.295936  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:01.425853  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:01.475879  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:01.487290  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:01.497615  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:01.498056  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:01.542686  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:01.578287  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:01.591826  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:01.594355  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:01.597208  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:01.613765  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:01.615464  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:01.644115  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 148556us
I20240503 02:18:01.647406  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:01.650592  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(62a3dcc2de594c8982477a0229f71191): perf score=0.033401
I20240503 02:18:01.662580  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:01.668125  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:01.682950  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:01.710139  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:18:01.879158  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.228s	user 0.043s	sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7045,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":9605,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7828,"threads_started":2,"wal-append.queue_time_us":525}
I20240503 02:18:01.880764  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191): 4923 bytes on disk
I20240503 02:18:01.881508  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(62a3dcc2de594c8982477a0229f71191) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20240503 02:18:01.890151  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:02.006466  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:02.020026  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:02.027410  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 50016.4us
I20240503 02:18:02.054718  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:02.059417  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:02.082728  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:02.091089  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:02.118585  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:02.175403  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:02.207034  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0): perf score=0.033344
W20240503 02:18:02.282712  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:02.291652  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.084s	user 0.046s	sys 0.034s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":33932,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":2617,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":862,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":41429,"threads_started":2,"wal-append.queue_time_us":7917}
I20240503 02:18:02.293452  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0): 4838 bytes on disk
I20240503 02:18:02.294095  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(9f7c9c29384949fb95cb0c3b765f57b0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 02:18:02.295128  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece): perf score=0.033343
W20240503 02:18:02.302249  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:02.402518  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: FlushMRSOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.107s	user 0.075s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":317,"dirs.run_wall_time_us":1952,"drs_written":1,"lbm_read_time_us":297,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1062,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10054,"threads_started":1,"wal-append.queue_time_us":10162}
I20240503 02:18:02.404358  4242 maintenance_manager.cc:419] P 1d36807868fe4a82bca42c25831ce8a2: Scheduling UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece): 4838 bytes on disk
I20240503 02:18:02.405494  4133 maintenance_manager.cc:643] P 1d36807868fe4a82bca42c25831ce8a2: UndoDeltaBlockGCOp(1ec063c94fe941ebb1c866b5b8935ece) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 02:18:02.484786  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:02.611768  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:02.638072  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:02.645819  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:02.690861  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:02.703564  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:02.719987  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:02.723974  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:02.726171  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:02.742266  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:02.742857  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:02.770412  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:02.775079  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:02.775735  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:02.786478  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:02.860193  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:02.868881  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:02.949427  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:02.988476  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:03.018880  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:03.077363  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:03.126242  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:03.128271  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:03.159984  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:03.163142  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:03.166649  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:03.182471  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:03.182945  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:03.325304  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:03.336426  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:03.516026  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling FlushMRSOp(771a4190142b4fd5a9479417122c0881): perf score=0.033374
W20240503 02:18:03.534562  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:03.554797  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 43811.1us
I20240503 02:18:03.567214  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:03.600229  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:03.607463  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:03.625726  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:03.658385  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:03.663270  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:18:03.671509  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:03.688323  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: FlushMRSOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.172s	user 0.036s	sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1032,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":6686,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1239,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":8810,"threads_started":2,"wal-append.queue_time_us":7557}
I20240503 02:18:03.692296  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881): 4581 bytes on disk
I20240503 02:18:03.692960  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 02:18:03.696995  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:03.709728  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:03.730851  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:03.735695  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:03.773624  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:18:03.777110  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:03.830746  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:03.961558  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:03.969846  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:03.980438  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:03.990294  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:03.995823  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:03.997059  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:04.001308  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:04.060746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:04.104661  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:04.127002  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:04.166745  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:04.213439  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:04.242763  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:04.543431  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:04.563344  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:04.579010  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:04.776002  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:04.831784  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:04.840310  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:04.875960  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:04.899693  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:04.942163  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:04.948069  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:04.974035  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:04.994534  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:05.027588  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:18:05.063624  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:05.069736  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:05.096815  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:05.121021  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:05.129688  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:05.176141  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:05.186895  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:05.190974  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:05.205492  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:05.217731  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:05.242341  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:05.279103  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:05.332317  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:05.335513  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:05.351423  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:05.391036  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:18:05.500644  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:05.663020  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:05.667822  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:05.708683  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:05.715759  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:05.748467  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:05.763227  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:05.774163  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:05.813719  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:05.842244  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:05.887946  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:18:05.944242  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:18:06.041781  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:06.058413  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:06.075578  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:18:06.077024  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:06.099380  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:06.111987  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:06.156227  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:06.172657  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:06.176030  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:06.201522  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:06.202538  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:06.222663  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:06.254530  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:06.297358  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:06.303584  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:06.318187  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:06.446672  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:06.547600  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:18:06.573257  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:06.673480  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:06.695089  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:06.707190  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:06.717476  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:06.721181  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:06.745223  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:06.763226  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:07.067934  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:18:07.098222  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:07.102981  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:07.131039  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:07.160849  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(734027e195414b15811ac9d32b132db8): perf score=0.033389
I20240503 02:18:07.181421  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:07.219506  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:07.221437  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:07.246754  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:07.280738  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:07.281960  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:07.333899  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:07.358399  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:07.358373  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:07.370219  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:07.384238  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.223s	user 0.045s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8721,"dirs.run_cpu_time_us":300,"dirs.run_wall_time_us":18683,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":967,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":71829,"threads_started":2,"wal-append.queue_time_us":59587}
I20240503 02:18:07.385972  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8): 4583 bytes on disk
I20240503 02:18:07.387907  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(734027e195414b15811ac9d32b132db8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 02:18:07.425603  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:07.441532  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:07.467350  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:07.476895  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:07.535558  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:07.563556  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:07.571467  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:07.582556  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:07.583381  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:07.605145  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:07.612710  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:07.624923  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:07.631070  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:07.656186  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:07.670676  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:07.826488  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:07.965988  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:08.072225  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:08.102443  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:08.110577  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:08.121268  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:08.140399  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:08.145022  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:08.147727  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:08.199987  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:18:08.218551  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342): perf score=0.033353
I20240503 02:18:08.243852  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:08.315793  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:08.371251  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.152s	user 0.071s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":570,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":3614,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1170,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10306,"threads_started":2,"wal-append.queue_time_us":9960}
I20240503 02:18:08.373389  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342): 4838 bytes on disk
I20240503 02:18:08.379855  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(eadcb49d12664e6fb5a5ad8dbe47f342) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240503 02:18:08.380908  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:08.422588  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:08.439482  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:08.456584  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:08.465303  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:08.507639  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:08.521627  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:08.549608  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:08.576140  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:08.577888  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:18:08.626083  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:08.627084  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:08.638813  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:08.643517  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:08.689534  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:08.708132  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:08.755321  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:08.813113  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:08.815016  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:08.828675  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:18:08.906730  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:18:09.055446  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling FlushMRSOp(586ebf467f0f427fa62dee6223a39848): perf score=0.033359
I20240503 02:18:09.068677  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:09.098609  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:09.118413  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:09.163654  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:09.163786  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:09.177073  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:09.182494  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:09.259987  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:09.273612  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:09.288054  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:09.289592  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:09.291124  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:09.302315  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:09.324965  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:09.368008  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:09.444593  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: FlushMRSOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.389s	user 0.046s	sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63705,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":3856,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":83289,"threads_started":2,"wal-append.queue_time_us":19591}
I20240503 02:18:09.450366  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848): 4923 bytes on disk
I20240503 02:18:09.451201  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(586ebf467f0f427fa62dee6223a39848) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240503 02:18:09.541402  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:09.545117  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:09.577172  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:09.581249  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:09.591506  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:09.593194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:09.632282  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:09.665091  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:09.745935  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:18:09.750763  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:09.937933  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:10.021605  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:10.030853  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:10.038149  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:10.089634  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:10.096884  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:10.138355  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:10.151002  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:10.158753  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:10.197191  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:10.211537  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:10.265712  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:10.267829  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:10.281520  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:10.298110  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:10.319967  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:10.354415  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:10.411665  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:10.435092  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:10.481824  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:10.507107  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:10.558320  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:10.574952  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:10.600375  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:10.603747  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:10.605594  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:10.673688  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:18:10.776363  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:10.880596  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:10.908962  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:10.909363  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:10.949488  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:10.951719  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:10.971540  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:11.008220  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:11.034574  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:11.055533  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:11.176399  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:11.213349  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:11.215340  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:18:11.247685  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240503 02:18:11.275743  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:11.303602  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:11.305833  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:11.312326  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:11.343297  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:11.357023  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:11.415086  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:11.435846  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:11.542966  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:11.556574  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:11.583783  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:11.610571  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:11.636564  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:11.650027  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:11.664563  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:11.668356  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:12.004079  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:12.006191  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:12.021627  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:12.027930  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:12.046788  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:12.082988  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:12.088837  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:12.127396  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:12.204432  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:12.208989  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:12.216434  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:12.292304  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:12.299825  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:12.303903  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:12.337608  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:12.349709  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:18:12.390477  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:12.391336  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:12.399842  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:18:12.535945  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:12.621366  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:12.628516  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:12.647624  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:12.649092  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:12.672158  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:12.685418  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:12.699775  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:12.706547  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:12.709811  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:12.734144  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:12.763001  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:12.779716  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:12.801229  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:12.834033  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:12.878094  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:12.893523  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:12.898919  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240503 02:18:13.190569  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:13.196276  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:13.197546  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:13.198312  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:13.200186  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:13.200922  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:13.236982  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:13.239763  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:13.314396  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:18:13.335209  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:18:13.526098  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:18:13.645624  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:18:13.647523  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:13.653856  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:13.686635  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:13.701043  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:13.717001  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:13.730563  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:13.770089  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:13.802758  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:13.858901  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:13.920527  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 201427us
I20240503 02:18:13.921033  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:13.922425  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:13.942162  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:13.963682  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:13.984791  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:14.006510  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:14.058766  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:18:14.071395  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:14.076813  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:14.078670  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:14.134395  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:18:14.168300  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:14.375922  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:14.390901  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:14.406515  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:14.417953  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:14.429098  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling FlushMRSOp(51224b6355ed41dbb0703b7feb73c223): perf score=0.033393
I20240503 02:18:14.443898  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:14.466189  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:14.466367  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:14.537070  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:18:14.586971  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:14.616103  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: FlushMRSOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.187s	user 0.049s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10803,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":5009,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1299,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18031,"threads_started":2,"wal-append.queue_time_us":6875}
I20240503 02:18:14.617395  4507 maintenance_manager.cc:419] P 408af0ab5520465e942cc714bfc6c73b: Scheduling UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223): 4838 bytes on disk
I20240503 02:18:14.618678  4398 maintenance_manager.cc:643] P 408af0ab5520465e942cc714bfc6c73b: UndoDeltaBlockGCOp(51224b6355ed41dbb0703b7feb73c223) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":352,"lbm_reads_lt_1ms":4}
W20240503 02:18:14.698449  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:14.746717  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:14.775903  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:14.786886  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:14.787453  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:14.802023  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:14.815136  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:14.846557  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:14.871814  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:15.131541  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:15.139434  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:15.139604  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:15.151676  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:15.186867  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:15.211071  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:15.224663  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:15.233963  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:15.271797  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:15.401683  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:18:15.401978  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:15.462661  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:15.481752  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:15.486105  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:15.495776  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:15.496194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:15.616182  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:15.641085  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:15.655681  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:15.683157  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:15.767443  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:15.829073  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:15.856428  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:15.920142  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:15.920395  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:15.940701  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:15.952370  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:15.980002  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:16.073856  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:16.180006  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:16.180006  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling FlushMRSOp(771a4190142b4fd5a9479417122c0881): perf score=0.033363
W20240503 02:18:16.206372  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:16.239085  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:16.265026  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:16.278673  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:16.327913  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:16.357040  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:16.413359  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:16.482905  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: FlushMRSOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.296s	user 0.077s	sys 0.042s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15557,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":10122,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1189,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58693,"threads_started":2,"wal-append.queue_time_us":43388}
I20240503 02:18:16.484879  2327 maintenance_manager.cc:419] P 08055479ea104aa290d19955d3720850: Scheduling UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881): 4581 bytes on disk
I20240503 02:18:16.485710  2218 maintenance_manager.cc:643] P 08055479ea104aa290d19955d3720850: UndoDeltaBlockGCOp(771a4190142b4fd5a9479417122c0881) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240503 02:18:16.533596  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:16.617273  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:18:16.627154  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:16.669816  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240503 02:18:16.688035  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:16.718860  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:16.745438  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:16.793000  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:16.823204  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:16.839740  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:16.866145  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:16.894001  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:16.922276  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:16.946072  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:16.990062  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:17.013180  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:17.056878  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:17.135577  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:17.167800  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:17.184979  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:17.322515  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:18:17.328492  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:18:17.373164  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:17.379228  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:17.384343  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:17.386216  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:17.387853  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:17.389081  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:17.389253  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:17.407833  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:17.537691  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:17.670523  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:17.681589  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:17.709885  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:17.726004  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:17.726719  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:17.728065  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:17.755074  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:18:17.817097  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:17.838205  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:17.840788  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:17.996878  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:18.006384  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:18.013712  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:18.014277  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:18.030020  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:18.057669  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:18.065687  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:18.067329  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:18.165360  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:18:18.210165  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:18:18.252724  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:18.309595  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:18.315446  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:18.348562  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:18.351948  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:18.356398  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:18.441776  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:18.462683  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:18.608752  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:18.666975  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:18:18.691411  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:18.692395  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:18.699970  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:18.703943  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:18.704401  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:18:18.709264  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:18.738451  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:18.744658  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:18.778821  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:18.794607  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:18.891130  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:19.003168  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:19.006877  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:19.013958  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:19.025051  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:19.026083  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:19.088172  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:19.100174  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:19.165768  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:18:19.360356  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:19.375170  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240503 02:18:19.392696  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:19.438475  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:19.457172  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:19.484548  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:19.579236  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:19.591802  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:19.597229  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:19.605914  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:19.679942  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:19.687585  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:19.721942  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:19.727238  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:19.782135  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:19.783615  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:19.789288  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:19.835939  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 55471.6us
W20240503 02:18:19.864723  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:18:19.915335  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:19.954275  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:20.033165  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:20.046721  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:20.063588  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:20.064911  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:20.078990  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:20.111702  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:20.144806  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:20.146023  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:20.259972  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240503 02:18:20.320681  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:20.338533  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:20.358479  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:20.380403  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:20.401757  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:20.402319  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:20.408879  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:20.469007  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:20.583623  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:20.599390  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:20.602595  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:20.613823  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:20.616482  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:20.642752  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:20.673705  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:20.771137  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:18:20.798671  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:18:20.879916  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:20.949600  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:20.961843  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:20.973646  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:20.981194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:20.982304  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:20.988513  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:21.036123  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:21.072391  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:21.074182  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:21.207870  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:21.229879  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:21.243830  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:21.253983  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:21.270771  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:21.280258  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:21.315667  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:21.321416  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:21.350139  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:18:21.503569  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:21.603368  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:21.625762  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:21.632730  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:21.650431  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:21.663028  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:21.689080  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:21.796517  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:21.857419  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:21.901990  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:18:21.903028  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:21.912053  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:21.930294  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:21.938246  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:21.992758  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:21.997062  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:22.037350  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:22.086364  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:22.137604  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:22.138777  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:22.147413  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:22.170571  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:22.192137  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:22.204550  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 61648.9us
I20240503 02:18:22.212929  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:22.239652  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:22.243387  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:22.327365  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:22.399910  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:22.460422  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:22.463639  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:22.489071  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:22.539484  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:22.539645  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:22.564153  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:22.570031  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:22.576189  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:18:22.639868  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:18:22.764007  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:22.784804  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:22.814527  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:22.845068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:22.863237  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:22.890275  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:22.916090  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:22.917255  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:18:22.922899  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:22.924680  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:22.956704  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:23.173436  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:23.175945  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:23.176070  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:23.188181  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:23.206478  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:23.224303  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:23.269739  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:23.331019  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:18:23.347867  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:23.442903  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:23.512135  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:23.518170  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 68475.1us
I20240503 02:18:23.530638  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:23.533082  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:23.545567  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:23.566426  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:23.573648  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:23.585103  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:23.646291  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 02:18:23.647253  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:23.679600  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:23.703249  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:23.740020  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:18:23.747445  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:23.748339  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:23.764859  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:23.821620  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:23.849747  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:23.874068  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:23.957712  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:23.970896  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:24.023286  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:24.033633  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:24.044486  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:24.044724  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:24.076464  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:24.101796  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:24.110594  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:24.226678  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:18:24.334303  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:24.345793  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:24.349056  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240503 02:18:24.355893  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:24.374625  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:24.382550  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:24.384686  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:24.417954  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:24.420689  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:24.548480  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:24.549985  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:24.590633  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:24.600402  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:24.615597  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:24.636494  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:24.684823  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:24.702044  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:24.757509  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 74684.3us
W20240503 02:18:24.757579  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 46582us
W20240503 02:18:24.896981  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:18:24.947880  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:25.077747  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:25.078449  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:25.119695  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:25.124960  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:25.131038  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:25.131884  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:25.134781  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:18:25.154042  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:25.172828  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:25.235260  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:25.323727  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:25.339004  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:25.355110  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:25.412757  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:25.415585  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:25.415690  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:25.416054  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:25.425666  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:25.477514  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:25.518803  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:25.562661  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:25.567581  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:25.636796  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:25.663394  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:25.669714  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:25.698565  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:25.830866  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:25.991192  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:26.002386  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:26.008338  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:26.017210  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:26.029418  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:26.035874  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:26.096467  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:26.135139  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:26.162357  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:18:26.221972  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:26.316740  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:26.343312  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:26.359335  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:26.362926  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:26.376336  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:26.388979  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:26.397537  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:26.399673  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:26.409518  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:26.424970  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:26.608984  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:26.617856  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:26.656280  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:26.668721  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:26.690668  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:26.722414  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:26.734247  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:26.838774  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:26.933364  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240503 02:18:26.982017  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:27.018566  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:27.061486  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:27.093748  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:27.129011  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:27.129442  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:27.145278  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:27.155086  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:27.164992  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:18:27.278718  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:27.288419  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:27.306633  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:27.336675  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:27.360360  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:27.384403  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:27.393728  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:27.429672  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:27.467399  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:27.474141  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:27.566313  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:27.575588  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:27.579799  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:27.595933  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:27.597645  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:27.598062  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:27.611794  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:18:27.637614  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:18:27.698786  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:18:27.777205  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:27.890151  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:27.897287  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:27.920518  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:27.924233  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:27.941475  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:27.942366  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:28.020790  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:28.023211  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:28.183498  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:18:28.201661  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:28.208797  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:28.224866  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:28.243547  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:28.274663  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:28.325083  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:28.327893  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:28.351277  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:28.435748  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:28.489864  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:28.495895  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:28.505805  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:28.518385  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:28.528293  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:28.534052  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:28.556983  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:28.712633  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:18:28.810389  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:28.823266  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:28.824692  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:28.832998  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:28.850495  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:28.866199  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:28.895663  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:28.895670  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:28.938673  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:28.971312  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:28.994675  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:29.024060  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:29.037184  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:29.150853  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:29.157775  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:29.160225  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:29.186524  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:29.198648  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:29.215018  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:29.224473  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240503 02:18:29.240947  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:29.327965  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:18:29.343516  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:29.345026  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:29.351146  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:29.367856  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:29.368391  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:29.397415  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:29.440439  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:29.440560  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:29.444979  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:29.650305  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:29.682500  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:29.710017  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:29.756175  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:29.802860  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:29.835273  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:29.850469  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:29.943145  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:30.009589  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:18:30.011461  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:30.029086  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:30.054780  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:30.062604  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:30.107602  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:30.144093  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:30.151722  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:30.160039  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:30.173028  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:18:30.188205  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240503 02:18:30.346840  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:30.386813  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:30.401742  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:30.418206  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:30.423038  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:30.442482  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:18:30.515416  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:30.529512  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:30.589417  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:30.727569  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:30.748351  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:30.755663  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:30.762935  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:30.774916  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:30.794982  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:30.803905  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:30.912225  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:30.992264  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:18:30.997638  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:31.008070  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:31.034986  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:31.052870  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:31.069638  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:31.071394  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:31.106599  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:31.192498  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:31.261361  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:31.269308  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:31.279430  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:31.281388  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:31.290971  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:31.326805  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:31.340174  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:18:31.358287  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:31.549021  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:31.558194  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:31.571514  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:18:31.605465  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:31.615037  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:31.615700  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:31.619318  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:31.660549  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:31.663336  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:31.693276  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:31.708528  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:31.725550  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:31.727437  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:31.735904  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:31.792217  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:31.848346  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:31.887281  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:31.891574  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:31.899942  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:31.996193  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:32.155206  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:32.165558  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:32.166594  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:32.176123  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:32.178391  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:32.192786  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:32.207800  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:32.392145  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:32.425457  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:32.444921  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:32.452657  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:32.459290  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:32.464836  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:32.484493  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:32.485805  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:32.488153  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:32.509132  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:32.603639  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:18:32.611523  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:32.630420  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:18:32.721690  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:32.727033  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:32.728631  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:32.745417  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:32.750015  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:32.758731  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:32.784507  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:32.790685  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:33.036118  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:33.040402  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:33.049778  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:33.065362  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:33.085961  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:33.102993  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:33.128260  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:33.130651  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:33.250079  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:33.267400  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:33.273120  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:33.301364  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:33.304828  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:33.309978  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:33.358353  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:33.407289  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:18:33.443162  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:33.520903  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:33.585419  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:33.612665  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:33.623370  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:33.662838  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:33.670336  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:33.723270  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:33.728128  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:33.756373  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:33.821763  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:33.823329  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:18:33.916283  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:33.917497  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:33.946899  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:18:33.968647  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:34.011703  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:34.012691  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:34.035789  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:34.040858  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:34.138819  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:34.138914  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:34.146060  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240503 02:18:34.152021  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:34.158421  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:34.216843  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:34.256798  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:34.259377  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:34.281263  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:34.297942  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:34.340363  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:34.351756  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:34.371639  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:34.380333  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240503 02:18:34.571888  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:34.573052  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:34.584046  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:34.585384  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:34.611544  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:34.635536  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:34.690830  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:34.695063  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:34.821799  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:34.963531  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:34.985855  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:35.033563  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:35.036404  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:35.065364  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:35.078850  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 93780.8us
I20240503 02:18:35.097247  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:35.099704  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:35.127671  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:35.140762  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:35.141541  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:35.146728  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:35.157214  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:35.181001  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240503 02:18:35.192643  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:35.263449  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:35.280021  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:35.298213  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:35.338092  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:35.388237  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:35.525817  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:35.542088  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:35.575637  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:35.600358  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:35.629863  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:35.686575  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:35.720026  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:35.857409  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:35.863310  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:35.885514  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:35.945407  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:35.957798  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:35.962715  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:35.963869  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:35.984191  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:36.077576  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:36.161613  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:36.200947  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:36.241535  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:36.251379  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:36.292706  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:36.321521  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:36.343781  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:36.363243  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:36.380178  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:36.490846  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:18:36.545392  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:36.558836  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:36.566233  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:36.593394  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:36.595422  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:36.600785  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:36.636627  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:36.646704  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:36.660418  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:36.678390  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:36.680933  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:36.939250  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:36.953438  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:36.959523  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:36.978695  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:37.014686  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:37.048183  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:37.081161  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:37.090068  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.240053  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:37.274327  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240503 02:18:37.276873  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:37.319442  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:37.333870  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:37.368582  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 69786.5us
W20240503 02:18:37.412668  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 120018us
I20240503 02:18:37.443320  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:37.450557  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:37.450795  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:37.451073  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:37.477293  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:37.534540  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:37.564954  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:37.622370  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:37.744426  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:37.759485  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:37.765637  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:37.776965  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:18:37.797472  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:37.856429  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:37.858928  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:37.895756  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:37.904726  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:37.917941  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.921545  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:37.965385  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:37.969633  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:38.014070  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:38.030251  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:38.100477  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:38.255553  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:38.261122  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:38.264540  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:38.272657  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:38.291633  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:38.301882  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:38.383086  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:38.508345  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:38.553597  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:38.562232  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:38.623401  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:38.650034  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:38.651291  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:38.656770  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:38.670864  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:38.705687  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:38.728719  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:38.728876  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:38.743228  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:38.759042  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:38.787987  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:38.833737  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:38.836614  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:38.893991  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:38.918716  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:38.921636  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:38.923337  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:38.943759  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:38.957566  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:38.981503  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:38.982575  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:18:39.111176  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:39.166920  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:39.169507  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:39.199951  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:39.213596  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:39.215584  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 57434.4us
I20240503 02:18:39.223096  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:39.225824  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:39.240907  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:39.387460  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:39.423419  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:39.427943  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:18:39.470634  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:39.497588  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:39.502005  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:39.502857  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:39.544634  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:39.560039  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:39.669911  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:39.677999  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 67487.4us
I20240503 02:18:39.800364  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:39.805246  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:39.825059  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240503 02:18:39.827061  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:39.835743  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:39.844843  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:39.876696  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:39.891564  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 68558.2us
I20240503 02:18:39.925942  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:39.928139  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:40.044692  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:40.048259  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:40.119081  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:40.128252  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:40.221675  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:40.237883  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:40.272193  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:40.299248  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:40.311442  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:40.340837  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:40.343196  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:40.369711  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:40.404788  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:40.404954  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:40.414464  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:40.428932  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:40.470681  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:40.487707  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:40.534081  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:40.736331  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:40.748394  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:40.767005  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:40.810235  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:40.857950  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:40.884972  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:40.889627  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:41.006855  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:41.020073  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:41.039968  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:41.049916  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:41.080405  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:41.108585  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:41.130470  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:41.133236  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:41.185985  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:41.275651  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:41.277119  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:41.279965  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:41.290045  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:41.291275  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:41.308709  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:41.311568  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:41.312685  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:41.363565  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:18:41.364642  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:41.510754  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:41.512956  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:41.622382  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:41.636471  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:41.670361  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:41.689003  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:41.690203  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:41.694142  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:41.726900  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:41.728354  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:41.772204  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:41.772688  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:41.782748  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:41.803697  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:41.825280  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:41.831956  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:41.852720  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:41.857771  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:41.864589  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 72397us
W20240503 02:18:42.063937  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:42.150606  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:42.157261  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:42.162593  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:42.187003  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:42.225980  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:42.252588  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:42.270774  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:42.386945  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:42.483870  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:42.490010  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:42.494210  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:42.545943  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:42.550395  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240503 02:18:42.564122  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:42.568128  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:42.569231  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:42.632493  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:42.641652  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:42.694137  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:18:42.745651  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:42.793210  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:42.815332  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:42.830514  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:42.858430  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:42.897064  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:42.942068  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:42.949743  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:42.961687  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 78392.8us
W20240503 02:18:42.962308  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:43.131623  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:43.152940  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:43.157382  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:43.172068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:43.191262  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:43.243628  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:43.267832  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:43.273965  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:43.335556  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:43.465127  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:43.474696  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:43.482688  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:43.502692  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:43.514256  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:43.549903  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:43.583813  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:43.694067  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:43.743983  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:18:43.789497  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:43.861538  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:43.887537  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:43.895699  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:43.902745  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:43.925266  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:43.938961  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:43.940186  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:43.950400  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:43.966292  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:43.967288  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:44.005992  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:44.010238  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:18:44.011467  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:18:44.070948  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:44.191634  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:44.195183  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:44.196365  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:44.229393  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:44.234161  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:44.248281  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:44.249768  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:44.273751  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:44.275557  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:44.298025  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:44.299454  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:44.329782  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:44.335912  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:44.340415  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:44.375391  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:44.431596  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:18:44.501384  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:44.632079  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:44.632767  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:44.640018  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:44.653712  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:44.661944  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:44.686990  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:44.688464  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:44.826810  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:18:44.919239  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:45.032290  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:45.034406  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:45.048624  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:45.060591  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:45.064438  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:45.080518  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:45.083639  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:45.115628  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:18:45.117424  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 50587.5us
I20240503 02:18:45.132158  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:45.145663  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:45.261508  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:45.266613  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:45.285941  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:45.305480  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:45.311053  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:45.351660  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:45.385520  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:45.419672  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:18:45.528849  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:45.597206  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:45.612561  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:45.646390  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:45.663971  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:45.782644  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:45.808691  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:45.871582  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:45.994457  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:46.126770  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:46.143484  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:46.164366  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:46.168033  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:46.203816  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240503 02:18:46.211593  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:46.222210  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:46.249406  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:46.259910  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:46.270977  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:46.284797  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:46.412714  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:46.419795  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:46.430169  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:46.442894  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:46.466336  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:46.499716  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:46.525998  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:46.531746  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:46.558686  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:46.760329  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:46.791519  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:46.792016  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:46.824569  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:46.827791  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:18:46.842398  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:18:46.872992  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:46.884027  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:46.893100  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:47.019383  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:47.085700  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:47.096537  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:47.138170  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:47.142303  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:47.159138  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:47.207046  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:47.208927  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:47.432224  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:47.442920  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:47.443457  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:47.460229  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:47.464056  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:47.481231  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:18:47.507130  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:47.563758  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:47.591013  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:47.603478  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:47.612275  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:47.647188  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:47.783116  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:47.795306  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:47.814090  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:47.823554  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:47.927845  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:47.957198  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:47.960943  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:47.963542  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:48.007128  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:48.026295  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 74563.3us
W20240503 02:18:48.029845  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:48.063448  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:48.069008  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:48.077642  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:48.129647  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:48.163890  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:48.186188  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:48.242300  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 83876.1us
I20240503 02:18:48.456182  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:48.459518  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:48.468552  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:48.482704  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:48.489535  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:48.506989  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:48.514734  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:48.526875  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:48.652896  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:48.655680  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:48.750826  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:18:48.847862  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:48.866262  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:48.884994  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:48.891522  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:48.891510  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:48.931111  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:48.935364  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:48.937464  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:48.952545  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:48.962169  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:49.003111  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:49.005493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:49.058606  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 54223.8us
W20240503 02:18:49.062161  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:49.066272  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 02:18:49.069609  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:49.069907  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:49.070159  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240503 02:18:49.076844  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:49.124276  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:49.129184  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:49.163396  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:18:49.167855  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:49.355969  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:49.362876  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:49.381986  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:49.387280  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:49.390583  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:49.407243  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:49.447927  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:49.463320  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:49.470583  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:18:49.480579  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:49.680155  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:49.704916  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:49.705694  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:49.718284  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:49.757833  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:49.778510  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:49.789165  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:49.898221  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:50.008179  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:18:50.014429  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:50.027477  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:18:50.033430  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:50.074755  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:50.093068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:50.097739  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:50.119161  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:50.120782  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:50.199541  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:50.205705  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240503 02:18:50.257212  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:50.262113  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:50.285156  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:50.297966  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:50.306047  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:50.320693  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:50.325665  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:50.334045  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:50.346920  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:50.589123  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:50.589203  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:50.607736  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:50.610299  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:50.610630  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:50.636945  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:50.645475  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:50.733891  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:50.947885  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:51.104072  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:51.149685  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:51.167426  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:51.177286  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:51.177521  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.202137  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:51.224180  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:51.234696  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:51.251109  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:51.269161  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:51.271737  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:51.276969  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.284391  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:51.289153  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:18:51.335161  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:51.346032  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:51.407116  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:51.427872  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:51.453275  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:51.466625  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:51.495580  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:51.510396  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:51.527834  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:51.528579  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:51.551065  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:51.552690  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:51.582823  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:18:51.762414  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:51.884461  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:52.095935  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:52.108840  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:52.127835  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:52.148923  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:52.168968  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:52.172384  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:52.204479  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:52.296417  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:52.333269  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:52.345276  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:52.374261  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:52.375125  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:52.386730  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:52.395834  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:52.409601  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:52.477412  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:52.484337  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:52.510748  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:52.513161  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:52.517983  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:52.543326  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:52.545784  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:52.587945  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:18:52.618702  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:52.620317  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:52.654604  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:52.691511  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:52.768622  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:52.881434  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:52.890599  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:52.905746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:52.922441  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:52.989496  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:53.002413  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:53.048851  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:53.163378  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:53.303444  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:53.303509  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:53.362048  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:53.375820  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:53.410145  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:18:53.424640  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:53.446936  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:53.488286  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 47122.7us
I20240503 02:18:53.509508  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:53.510708  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:53.536091  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:53.542224  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:53.551542  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:53.582332  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:53.587582  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:53.625718  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:53.670953  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:53.692940  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:53.777451  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:53.815953  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:53.872532  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240503 02:18:53.913415  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:53.921725  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:53.927500  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:53.937691  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:18:53.945657  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:53.963279  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:53.963897  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:54.019011  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:54.073602  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:54.087482  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:18:54.090847  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:54.159587  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:18:54.160672  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:54.331898  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:54.359483  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 45061.9us
I20240503 02:18:54.379551  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:54.436750  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.447582  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:54.459987  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:54.501540  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:54.517799  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:18:54.518930  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:54.542646  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:54.577828  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:54.611455  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:54.616653  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:54.651580  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:54.670014  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:54.694175  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:54.716751  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:54.738960  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:54.772118  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:54.773840  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:54.776873  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:54.785472  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:54.832319  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:54.847751  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:18:54.923978  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:54.928036  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:54.936748  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:55.008543  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:18:55.196882  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:18:55.217021  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:55.233857  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:18:55.299022  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:55.314930  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.326637  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:55.347728  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:55.350441  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:55.356168  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:55.417218  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:55.658268  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:55.670887  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:55.716225  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:55.717867  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:55.719959  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:55.721407  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:55.724300  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:55.779116  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:55.908435  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:56.014497  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:56.041093  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:56.059489  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:56.088749  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:56.097982  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 50666.3us
W20240503 02:18:56.120802  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:56.126456  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:56.127586  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:56.150449  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:56.208004  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:56.271313  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:56.384960  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:56.387750  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:56.388393  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:56.409119  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:56.417572  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:56.420097  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:56.435377  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:56.449364  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:56.459031  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:56.462160  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:56.486166  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:56.507416  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:56.533233  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:56.559270  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:56.571655  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:56.577230  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:56.683332  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:18:56.730703  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:56.730902  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240503 02:18:56.901170  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:56.920185  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:56.950961  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:56.975997  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:56.976267  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:56.991150  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:56.993304  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:57.204255  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:57.318699  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:57.346829  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:57.392612  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:57.399417  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:57.440649  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:57.499290  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:57.529050  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:57.557535  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:57.566455  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:57.566481  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:57.574931  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:57.575325  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:57.610352  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:57.648247  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:18:57.659137  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:57.691922  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:18:57.712046  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:57.735816  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:57.861109  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:58.024788  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:18:58.054378  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:58.056753  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:58.060901  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:58.065531  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:58.065994  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:58.070798  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:58.085681  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:58.086120  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:58.303848  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:58.321942  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:58.343838  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:58.372875  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:58.373605  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:18:58.374794  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:58.406072  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:58.471737  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:18:58.548270  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:18:58.555989  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:58.585911  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:58.634112  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:58.644668  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:58.654287  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:58.670389  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:58.681145  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:58.688966  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:58.708348  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:58.903326  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:18:58.919495  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:18:58.948963  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240503 02:18:59.014220  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:59.028347  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:59.042654  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:59.049882  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:59.057850  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:59.080525  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:18:59.111222  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:18:59.132138  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:59.137542  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:18:59.208770  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:59.222486  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:18:59.236785  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:59.241192  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:18:59.258934  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:59.262667  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:59.281626  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:59.318799  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:59.355674  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:18:59.379276  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:59.585557  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:18:59.591430  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240503 02:18:59.657640  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:18:59.666352  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:18:59.675129  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:18:59.684396  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:18:59.719492  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:18:59.796204  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:18:59.867393  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:18:59.867611  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:18:59.902019  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:18:59.969235  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:00.053376  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:00.077816  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:00.079708  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:00.084631  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:00.102942  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:00.127732  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:00.138414  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:00.163514  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:00.172724  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:00.239799  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240503 02:19:00.325567  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:00.325911  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:00.339500  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:00.350122  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:00.353634  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:00.364612  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:00.391388  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:00.653219  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:19:00.729743  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:00.892174  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:00.897933  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:00.898957  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:00.972740  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:00.993925  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:01.054594  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:01.055778  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:01.084784  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:01.139248  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:01.148049  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 91045.4us
I20240503 02:19:01.168795  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:01.177003  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:01.187583  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:01.227465  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:19:01.233186  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:01.301786  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:01.320046  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:01.328886  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:01.335963  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:19:01.372182  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:19:01.422067  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:19:01.424973  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:01.700604  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:01.708961  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:01.718828  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:01.727497  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:01.727828  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:01.750073  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:01.752983  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:01.809051  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:01.871973  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:01.878361  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:01.888206  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:01.961208  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:01.967013  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:01.977716  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:01.991143  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:02.059280  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:02.197304  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:02.198661  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:02.207911  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:02.219094  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:02.244321  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:02.255985  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:02.258915  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:02.265825  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:02.337213  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:19:02.477535  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:19:02.543104  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:02.667241  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:02.667371  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240503 02:19:02.684378  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:02.702068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:02.738447  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:02.773942  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:02.780081  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:02.782276  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:02.782984  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:02.803092  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:02.811954  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:02.834584  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:02.854251  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:02.939508  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:02.939851  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:02.949298  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 45145.9us
I20240503 02:19:02.992100  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:03.250845  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:03.257804  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:03.263670  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:03.280728  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:03.284696  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:03.292287  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:19:03.316154  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:03.335867  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:03.343470  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:03.348492  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:03.358289  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:03.373644  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:03.431468  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:03.444911  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:03.467674  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:03.502002  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:03.527617  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:03.739907  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:03.740165  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:03.753021  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:03.756570  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:03.771075  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:03.821282  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:03.838397  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:03.843999  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:03.879456  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:03.880904  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:03.912660  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:19:03.955211  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:04.151276  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:04.153718  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:04.159868  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:04.171870  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:04.174350  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:04.200598  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:04.202955  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:04.207893  4132 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 45151.7us
I20240503 02:19:04.209460  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:04.231168  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:04.259168  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:04.283402  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:04.289093  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:19:04.306989  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:04.313014  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:04.321450  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:04.335335  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:04.364377  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:19:04.406594  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:19:04.408283  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:04.438208  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:04.621989  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:04.635870  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:04.650312  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:04.689463  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:04.709693  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:04.745572  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:04.763993  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:04.807219  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:04.836339  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:04.858091  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:04.862390  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:04.870332  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:19:04.893287  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:04.901963  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:04.929594  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:04.930994  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:05.013025  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:05.023757  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:19:05.045742  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:05.047978  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:05.145217  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:05.146245  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:05.148101  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:05.160739  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:05.174571  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:05.194487  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:05.194597  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:05.195318  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:05.512629  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:05.554333  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:05.557107  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:05.578857  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:05.614614  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:05.615823  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:05.621707  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:05.628618  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:05.629832  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:05.674130  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:05.699877  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:05.706784  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:05.717650  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:05.783202  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:05.814745  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:05.860044  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:19:05.900779  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:06.060302  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:06.062393  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:06.079964  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:06.087657  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:06.103570  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:06.104808  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:06.121686  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:06.164736  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:06.169430  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:06.170047  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:06.232609  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:06.339066  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:06.347419  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:06.356356  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:06.400189  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:06.437399  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:06.442028  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240503 02:19:06.467299  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:06.476877  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:06.497974  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:06.704115  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:06.711158  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:06.718058  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:06.719537  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:06.761210  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:06.777973  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:06.791389  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:06.809615  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:06.824168  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:06.930016  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:07.108883  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:07.117091  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:07.140043  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:07.140024  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:07.145696  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:07.164415  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:07.165778  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:07.265163  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:07.365103  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:07.365429  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:07.378563  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:07.382385  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:07.382454  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:07.409142  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:07.413259  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:07.441767  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:19:07.492440  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 50074.2us
W20240503 02:19:07.498106  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 58868us
W20240503 02:19:07.510437  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:07.685043  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:19:07.797472  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:07.802556  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:07.808306  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:07.815451  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:07.843374  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:07.973655  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:07.980093  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:08.035426  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:08.052194  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:08.078150  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:08.087095  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:08.123286  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:08.136394  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:08.154347  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:08.173313  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:08.207254  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:08.217962  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:08.259511  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:08.272253  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:08.304808  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:19:08.310313  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:08.336228  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:08.357517  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:08.415021  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:08.415529  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:08.436677  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:19:08.604151  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:08.763785  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:08.765674  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:08.798336  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:08.801919  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:08.817967  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:19:08.836798  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:08.857587  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:08.877348  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:08.952493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:09.098701  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:09.100374  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:09.104622  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:09.125172  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:09.134608  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:09.137504  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:09.151295  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240503 02:19:09.196238  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:09.196558  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:09.207480  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:09.243196  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:09.421967  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:09.427393  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:09.439075  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:09.451308  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:09.487100  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 59703.1us
I20240503 02:19:09.499596  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:09.533066  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240503 02:19:09.545408  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:09.546427  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:09.552131  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:09.558857  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:19:09.667466  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:09.767554  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:09.817626  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:09.848960  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:09.855533  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:09.885602  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:09.954413  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:09.955529  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:09.979144  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:09.999992  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:10.029076  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:10.039569  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:10.048755  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:10.064042  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:19:10.065271  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:10.083568  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:10.129386  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:10.154141  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:10.161851  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:10.176043  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:10.271281  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:10.434762  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:10.435832  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:10.458568  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:10.463066  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:10.472599  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.474941  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:10.488029  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:10.559140  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:10.674329  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:19:10.752825  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:10.789132  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:10.791357  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:10.814771  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:10.838642  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:10.878269  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:10.895035  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:10.947161  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:11.016927  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:11.025024  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:11.032395  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:11.054762  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:11.071189  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:11.072355  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:11.080737  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:11.166786  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:11.166923  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:11.222790  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:11.321419  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:11.337208  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:11.355906  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:11.366360  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:11.388185  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:11.393316  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:11.405220  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:11.449649  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:11.457024  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:11.529109  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240503 02:19:11.653079  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:11.654109  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:11.677659  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:11.700095  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:11.740891  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:11.783383  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:11.814213  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:11.899855  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:11.903860  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:19:11.968652  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:12.065651  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:12.101294  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:12.110881  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:12.118042  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:12.144112  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:12.154480  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:12.174175  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:12.189841  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:12.192186  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:12.325206  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:12.344892  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:12.348310  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.358749  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:12.362231  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:12.400290  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:12.416558  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240503 02:19:12.417182  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:12.472417  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:12.614307  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:12.617794  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:12.637030  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.643873  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:12.694590  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:12.734782  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:12.744616  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:12.757761  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:12.913193  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:12.920424  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:12.941239  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:12.961409  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:12.990393  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:13.000766  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:13.023144  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:13.024082  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:13.237319  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:13.264056  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:13.264369  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:13.266626  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:13.316267  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:13.332369  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:13.344899  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:13.371292  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:13.397656  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:13.416204  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:13.418931  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:13.430961  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:13.450711  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:13.472965  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:13.480970  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:13.529671  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:13.542402  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:13.668264  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:13.730943  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:13.736579  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:13.739300  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:13.754940  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:13.761022  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:13.768847  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:13.793828  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:13.807921  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:13.901396  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:19:13.957696  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:14.146405  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:14.200742  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:14.208796  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:14.209627  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:14.220068  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:14.234582  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:14.240764  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:14.266947  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:14.268981  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:14.294992  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:14.298884  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:14.313699  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:14.359570  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:14.360234  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:14.365370  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:14.381708  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 111431us
I20240503 02:19:14.381989  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:14.493254  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:19:14.493968  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:14.503268  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:19:14.565991  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:19:14.568328  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240503 02:19:14.660419  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:14.685046  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:14.709996  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:14.720043  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:14.793605  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:14.856559  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:14.869503  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:14.897166  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:19:14.915107  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:14.920193  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:14.932832  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:14.952354  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:14.961091  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:14.975545  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:15.006728  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:15.008883  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:15.046204  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:19:15.055294  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:15.178568  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:15.216238  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:15.230477  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:15.237653  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:15.239246  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:15.244741  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:15.246477  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.266170  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:15.312353  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:19:15.584473  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:15.628291  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:15.631757  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:15.633580  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:15.650857  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:15.655211  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.677315  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:15.708453  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:15.749516  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:15.750772  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:15.754765  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:15.757987  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:19:15.762024  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:15.765194  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 43610.3us
W20240503 02:19:15.807257  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:15.809841  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:15.850350  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:15.886875  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:15.973526  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:16.153970  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:16.175273  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:16.179033  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.190382  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:16.211870  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:19:16.222636  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:16.223068  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:16.250100  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:16.270249  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:16.418599  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:16.419559  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:16.438530  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:16.497181  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:16.498818  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.578471  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:16.589547  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:16.642736  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:16.659657  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:19:16.662235  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:19:16.750497  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:19:16.791033  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:16.794564  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:16.832316  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:16.854760  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:16.861582  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:16.916740  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:16.980250  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:16.988868  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:17.089848  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:19:17.164208  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:17.199318  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:17.208282  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:17.229933  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:17.237170  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:17.243281  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.254987  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:17.257565  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:17.269770  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:17.356588  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:17.420344  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:17.450817  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:17.455109  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.457067  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:17.471778  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:17.505530  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:17.523092  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:17.642796  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:17.653101  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240503 02:19:17.742461  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:17.765159  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:17.804857  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:17.843405  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:17.917928  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:17.960502  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:17.975399  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:18.008839  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:19:18.023242  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:18.108144  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:18.134217  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:18.152630  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.181633  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:18.207850  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:18.242369  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:18.293939  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:18.310854  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:18.389704  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:18.442771  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:18.448115  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:18.457371  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.460774  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:18.483479  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:18.489264  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:18.527561  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:18.549962  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:18.692203  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:18.694871  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:18.719400  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:18.720507  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:18.737511  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:18.743443  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:18.765973  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:18.796089  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:18.906775  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:19:18.991492  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:19.023397  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:19.053730  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:19.092557  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:19.119374  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:19.120123  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:19.120630  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:19.175837  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:19.176939  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:19.252815  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:19:19.279687  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:19.466075  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:19.487588  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:19.500707  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:19.528560  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:19.538239  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:19.580503  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:19:19.585948  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:19.605676  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:19.620945  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:19.650508  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:19:19.675060  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:19.803520  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:19.810295  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:19.815140  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:19:19.829412  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:19.866405  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:19.866614  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:19.929204  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:19.931641  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:19.948146  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:20.034381  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:19:20.035333  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:20.111047  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:20.118450  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:20.120556  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:20.139549  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:20.163607  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:20.182755  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:20.191687  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:20.214506  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:19:20.347680  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:19:20.358526  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:19:20.369426  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:20.379220  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:20.417762  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:20.453298  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:20.454186  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:20.458626  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:20.470050  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:20.494405  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 88516.2us
W20240503 02:19:20.504776  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 63958.2us
W20240503 02:19:20.644454  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:20.739351  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:20.787959  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:20.835809  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:20.841919  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:20.854431  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:20.880121  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:20.937594  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:20.949885  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:20.973981  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:20.981580  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:21.034821  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:21.066269  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:21.104338  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:21.118932  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:21.133515  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:21.150795  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:21.155858  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:21.223809  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:21.269814  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:19:21.288998  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:21.334856  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:21.342816  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:21.360864  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:21.386489  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:21.405606  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:21.447333  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:21.479207  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:21.617822  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:19:21.662442  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:21.711458  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:21.724284  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:21.748829  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:21.760392  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:21.803416  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:21.808998  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:21.820600  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240503 02:19:21.862367  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:22.039530  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:22.049860  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:22.068418  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:22.069185  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:22.070694  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:22.073606  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:22.075328  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:22.093605  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:22.161295  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:22.198738  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:19:22.256047  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:22.300935  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:22.304453  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:22.328954  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:22.329227  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:22.347057  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:22.347057  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:22.389019  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:22.403911  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:22.605738  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:22.611980  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:19:22.618247  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:22.646782  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:22.665625  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:22.670356  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:22.673611  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:22.685561  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:22.704885  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:22.709141  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:23.003566  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:23.038271  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:23.118978  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:23.176216  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:23.191066  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:23.229223  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:23.246258  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:23.260035  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:23.271922  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:23.328073  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:23.359854  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:23.394447  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:23.438206  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:23.479980  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:23.484563  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:23.510860  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:23.536430  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:23.546341  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:23.562492  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:23.568434  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:23.568491  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:23.607244  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:23.613035  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:23.659925  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:23.681979  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:23.683133  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:23.863554  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:19:23.863596  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:23.945199  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:23.959794  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:23.962096  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:23.980688  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:24.026654  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:24.053997  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:24.063239  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:24.125375  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:24.284293  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:24.310916  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:24.310986  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240503 02:19:24.333109  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:24.351125  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:24.389607  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:24.417322  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:24.435083  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:24.438802  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:24.461704  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:24.467415  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:24.518647  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:24.524878  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:24.548480  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:24.570675  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:24.596149  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:24.630628  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:19:24.675554  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:19:24.676627  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:19:24.731153  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:19:24.792732  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:19:24.855964  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:24.906242  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:24.943722  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:24.943955  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:24.954666  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:24.987402  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:25.029193  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:25.033084  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:25.053293  4132 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 57928us
W20240503 02:19:25.105859  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:19:25.107628  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:25.226202  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:25.233685  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:25.240280  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:25.242306  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:25.242348  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:25.269677  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:25.294931  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:25.307066  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:25.328555  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:25.495417  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:25.513715  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:25.520074  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:19:25.522096  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:25.530375  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:25.531124  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:25.540686  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:25.554673  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:25.581219  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:25.605607  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:25.700965  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:25.799119  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:25.835734  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:25.836241  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:25.872907  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:25.899382  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:25.918987  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:25.966998  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:25.972324  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:26.067210  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:26.165876  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:26.178321  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:26.183018  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:26.192989  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:26.198702  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:26.227695  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:26.269399  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:26.318128  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:26.436705  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:26.456398  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:26.467939  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:26.482513  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:26.498045  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:26.525674  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:26.565549  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:26.565721  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240503 02:19:26.602669  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:26.622465  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:26.699741  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:26.808732  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:26.846549  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:26.855185  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:26.907331  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:26.908800  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:26.928956  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 57864.4us
I20240503 02:19:26.961724  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:26.963711  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:26.964875  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:26.967943  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:26.993008  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:27.005064  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:27.014472  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:27.033660  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:27.061563  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:27.083693  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:27.117604  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:27.136919  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:19:27.182053  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:19:27.315917  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:27.343008  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:27.345579  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:27.353587  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:27.384337  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:27.397877  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:27.400771  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:27.412618  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:27.437115  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:27.735688  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:19:27.760428  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:19:27.800402  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 48582.1us
I20240503 02:19:27.821231  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:27.849272  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:27.850565  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:27.869009  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:27.885411  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:27.897512  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:27.925213  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:27.928372  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:27.934512  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:19:27.947084  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:28.006541  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:28.007391  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:28.019073  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:28.062657  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:28.101745  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:28.147019  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:28.169672  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:28.199589  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 57211.5us
I20240503 02:19:28.372743  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:28.403720  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:28.440096  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:28.442751  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:28.465561  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:28.489771  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:28.555191  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:28.583503  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:28.622992  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:28.682060  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:28.682422  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:28.712695  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:28.714645  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:28.736763  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:28.758633  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:28.760689  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:28.889863  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:29.022785  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:29.033211  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:29.035048  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:29.070794  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:29.078449  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:29.089874  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:29.134411  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240503 02:19:29.152704  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:29.198645  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:19:29.315271  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:29.323016  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:29.324051  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:29.341109  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:29.373139  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:29.379761  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:29.423772  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:29.426018  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:29.479488  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:29.495715  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:29.727480  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:29.735082  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:29.739053  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:29.774374  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:19:29.796262  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:29.809716  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:29.838099  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:29.856917  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:29.874497  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:29.917034  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:29.957257  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:19:29.982972  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:19:30.021906  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:19:30.141309  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:30.154605  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:30.158730  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:30.196027  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:30.215250  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:30.237505  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:19:30.242383  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:30.251610  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:30.282761  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:30.316300  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:30.357401  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:30.384943  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:30.556842  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:30.560608  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:30.593405  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:30.610116  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:30.629936  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:30.647300  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:30.658787  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:30.677816  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:30.690495  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:30.695315  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:19:30.860570  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:30.883642  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:30.896843  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:30.908291  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:30.913867  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:30.926395  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:30.955953  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:31.076179  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:31.195299  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.197120  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:31.204018  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.220453  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.248621  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:31.302037  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.302042  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:31.343396  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:31.409801  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:31.577236  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:31.619395  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:31.630111  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:31.687827  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:31.707835  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:31.718784  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240503 02:19:31.719903  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:31.724170  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.773550  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:31.801939  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.823261  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:31.842991  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:31.853258  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:19:31.881645  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:31.928756  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:31.973392  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:31.982228  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:32.019992  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:32.061573  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:32.219111  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240503 02:19:32.269071  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:32.279382  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:32.281265  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:32.314119  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:32.316556  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:32.383163  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:32.389894  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:32.444561  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:32.450767  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:19:32.516443  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:32.648188  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:32.732079  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:32.744782  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:32.748440  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:32.769445  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:32.782754  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:32.786935  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:32.794709  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:32.831856  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:32.872097  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:33.036522  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:33.060655  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:33.069455  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:33.107646  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:33.109099  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:33.127537  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:33.166293  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:33.192757  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:33.226220  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:19:33.243471  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:33.329093  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:33.355537  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:33.357705  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:33.358201  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:33.386699  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:33.403856  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:33.454816  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:33.718719  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:33.743878  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:33.745771  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:33.757220  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:33.759357  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:33.774286  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:33.796468  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:33.848280  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:33.934681  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:34.039557  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:19:34.059598  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:34.140296  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:34.172648  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:34.190778  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:34.204555  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:34.205415  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:34.253299  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:34.256592  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:34.290287  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:34.410065  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:34.425606  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:34.464468  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:34.517880  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:34.522403  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:34.557735  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:34.576897  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:34.577364  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:34.638535  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:34.723559  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:34.733587  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:34.734014  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:34.753262  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:34.767309  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:34.795637  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:34.797127  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:34.803485  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:19:34.813597  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:34.867048  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:19:34.886094  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:34.993837  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:35.074724  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:35.086583  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:35.109445  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:35.143703  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:35.159385  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:35.174609  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.206256  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:35.207715  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:35.277101  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:19:35.310454  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:19:35.399931  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:35.402753  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:35.419334  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:35.421128  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:35.441840  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:35.451435  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.477041  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:35.495690  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:35.509687  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:35.585389  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:19:35.669909  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:35.728578  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:35.736897  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:35.774775  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:35.780548  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:35.782438  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:35.803937  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:35.834178  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:35.835004  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:19:35.930271  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:36.066347  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:36.087343  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:36.121173  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:36.158080  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:36.176441  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:36.216028  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:36.235193  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:36.255918  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:36.422401  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:36.474393  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:36.489861  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:36.513638  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:36.518431  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 51156.4us
W20240503 02:19:36.528970  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:36.538569  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:36.582134  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:36.603672  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:36.638418  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:36.640172  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:36.643389  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:36.688622  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:36.720170  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:36.724880  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:36.730325  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240503 02:19:36.749291  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:36.785110  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:36.802691  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:36.810690  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:37.062494  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:37.083247  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:37.085558  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:37.086411  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:37.103864  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:37.120291  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:37.133476  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:37.154879  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:37.267630  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:19:37.379143  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:37.407963  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:19:37.410812  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:37.502941  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:37.528146  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:37.545703  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:37.576435  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:37.618873  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:37.633412  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:37.653213  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:37.687983  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:37.703204  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:37.703256  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:37.708616  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:37.749846  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:37.766355  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:37.787926  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:37.827003  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:19:37.835085  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:37.917474  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:37.943024  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.112598  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:38.137034  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:38.146950  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:38.159010  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:38.186841  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:38.263075  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:19:38.274838  4132 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 104072us
I20240503 02:19:38.290671  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:38.301724  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:38.315297  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:38.327507  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:38.327947  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:38.496362  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:38.504544  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:38.506531  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:38.530716  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:38.532869  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.594300  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:38.604161  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:38.827950  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:38.859422  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:38.866138  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:38.885429  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:38.912528  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:38.943470  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:38.999732  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:39.003472  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:39.004715  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:39.220686  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:39.231288  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:39.237989  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:39.273006  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:39.309863  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:39.321674  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:39.337772  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240503 02:19:39.355341  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:39.432050  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:19:39.546552  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:39.615162  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:39.637930  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:39.646061  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:39.682780  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:19:39.719934  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:39.732527  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:39.781466  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:39.797585  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 151769us
I20240503 02:19:39.875412  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:39.877287  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:39.895300  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:19:39.896385  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:40.034636  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:40.043323  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:40.058401  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:40.075021  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:40.097606  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:40.113533  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:40.125715  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:40.144522  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:40.168301  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:40.189586  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:40.355876  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:40.372239  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:19:40.399317  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:40.413156  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:40.413482  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:40.454046  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:40.471168  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:40.479122  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:40.509904  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:40.561930  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:19:40.630920  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:40.661237  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240503 02:19:40.720525  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:40.766579  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:40.766628  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:40.792712  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:40.807178  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:40.809880  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:40.812971  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:40.872622  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:40.874960  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:40.983316  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:19:41.107867  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:41.107895  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:41.108068  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:41.135237  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.150866  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:41.175480  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:41.212846  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.214331  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:41.381712  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:41.390486  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:41.409979  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.420621  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:41.437948  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:41.443881  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:41.462462  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:41.626680  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:41.632141  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:41.644194  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:41.652487  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:41.661290  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:41.695511  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:41.708637  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:41.711633  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:41.746870  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 62998.8us
W20240503 02:19:41.758113  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:19:41.759399  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:19:41.869871  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:19:42.017257  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:19:42.086177  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:42.158408  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:42.182283  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:42.245213  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:42.266614  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:42.278236  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:42.289240  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:42.338773  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:42.377597  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:42.389199  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:42.397439  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:42.404224  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:42.447244  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:42.469419  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:42.479396  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:42.504657  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:42.507172  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 124649us
I20240503 02:19:42.519567  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:42.529525  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:42.548079  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:42.548918  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:42.568255  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:42.584312  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:42.599490  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:42.605170  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:42.633656  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:42.642336  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:42.662846  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:42.682684  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:42.736379  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:43.148699  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:43.167716  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240503 02:19:43.223748  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:43.263293  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:43.263371  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:43.269034  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:43.272357  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:43.329577  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:43.336525  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:43.342941  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:43.464886  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:43.485265  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:43.492676  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:43.511184  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:43.560196  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 68720us
I20240503 02:19:43.575608  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:43.579424  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:43.619448  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:43.620451  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:43.704814  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:43.709918  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:43.727720  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:43.730294  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:43.735733  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:43.767863  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:43.797857  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:43.801913  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:44.128805  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:44.129351  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:44.133829  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:44.145044  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:44.164795  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:44.166102  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:44.167264  4129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 47887.6us
W20240503 02:19:44.223850  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:44.290478  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:44.330094  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:44.350487  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:44.370878  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:44.402381  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:44.425549  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:44.439467  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:44.441615  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:44.483510  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:44.500006  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:19:44.521102  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:19:44.599942  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:19:44.650470  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:19:44.659274  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:44.686756  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:44.691136  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:44.710813  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:44.720330  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:44.732781  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:44.772501  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:44.783591  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:19:44.865739  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:44.911999  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:19:44.913396  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:19:45.039533  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:45.057930  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:45.079640  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:45.087787  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:45.106464  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:45.118619  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:45.145495  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:45.187677  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:45.275394  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240503 02:19:45.284823  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:45.312263  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:45.313390  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:45.326975  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:45.351473  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:45.373482  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:45.379462  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:19:45.383534  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:45.474606  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 61417.1us
W20240503 02:19:45.515309  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:19:45.639667  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:45.644361  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:45.650954  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:19:45.672708  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:45.698640  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:45.717608  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:45.737913  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:45.741638  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:45.751449  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:45.770295  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:45.850347  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:45.883705  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:45.917959  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:45.944316  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:45.969219  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:45.992873  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:46.050490  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:46.092141  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:46.092615  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:46.233100  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 02:19:46.236183  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:46.310490  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:46.324713  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:46.343199  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:46.343451  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:46.398594  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:46.432122  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:46.441895  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:46.571748  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:46.581712  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:46.598953  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:46.623955  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:46.651202  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:46.699934  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:46.704553  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:46.751245  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:46.818061  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:46.842852  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:46.845523  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:46.940842  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:19:47.050415  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:47.077248  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:47.085979  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:47.088714  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:47.092352  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:47.110690  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:47.111441  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:47.148630  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:47.150223  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:19:47.189189  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:47.206789  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:47.228672  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:47.264462  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:47.273622  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:47.295897  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:47.335515  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:47.347112  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:47.431488  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:47.607265  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:47.626055  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:47.634166  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:47.650413  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:47.667569  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:47.667989  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:47.698057  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:47.711632  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:47.727218  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:47.746600  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:47.762696  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240503 02:19:47.781608  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:47.793287  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:47.811314  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:47.835023  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:47.838721  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:47.855199  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:47.920713  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 68761.4us
W20240503 02:19:47.954766  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:19:48.084636  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240503 02:19:48.102437  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:48.104065  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:48.127846  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:48.128206  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:48.138278  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:48.140805  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:48.144919  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:48.193902  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:19:48.242828  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:48.271458  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:48.438118  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:48.468959  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:48.490254  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:48.497293  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:48.497712  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:48.522421  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:48.564664  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:48.647629  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:48.673004  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:48.688482  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:48.698546  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:48.703997  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:48.738037  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:48.746196  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:48.750207  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:49.039573  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:49.050472  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:49.062124  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:49.063330  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:49.079756  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:49.089748  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:49.096850  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:49.125964  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:49.148612  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:19:49.223974  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:49.298866  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:49.322084  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:49.340066  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:49.378168  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:49.383555  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:49.400832  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:49.413193  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:49.420810  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:49.440768  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:19:49.596459  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:19:49.608636  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:49.624310  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:49.633546  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:49.639622  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:49.645274  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:49.667855  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:49.697563  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:49.779387  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:49.814595  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:49.896135  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:49.907493  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:49.950472  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:49.951325  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:19:49.952654  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:49.962066  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:49.964136  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:50.001518  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:50.024353  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:50.180076  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:50.195124  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:50.206061  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:50.217955  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:50.221232  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:50.242748  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:50.262717  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:50.276716  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:50.651180  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:50.660558  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:50.671142  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:19:50.680461  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:19:50.681416  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:50.682358  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:50.723431  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:19:50.728576  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:19:50.737073  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:50.750411  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:50.803714  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:50.815047  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:50.863035  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:50.863265  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:50.863840  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:50.865681  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:50.947098  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:50.953150  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:19:51.007550  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 208469us
I20240503 02:19:51.029948  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:51.054980  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:51.056748  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:51.074399  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:51.081161  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:51.102123  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:51.119169  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:51.141484  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:51.157500  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:51.162457  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:51.233143  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240503 02:19:51.250010  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:51.377025  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:51.670704  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:19:51.691147  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:51.715200  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:51.723017  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:51.811257  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:51.822993  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:51.870117  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:51.874241  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:51.895174  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:51.915938  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:52.011951  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:52.052438  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:52.057010  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:52.075822  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:52.087642  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:52.129611  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:52.157904  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:52.202188  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:52.202808  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:52.240753  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:52.324558  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240503 02:19:52.361433  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:52.362793  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:52.384996  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:52.444281  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 50678.3us
I20240503 02:19:52.450449  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:52.479113  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:52.489624  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:52.490152  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:52.556138  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:19:52.591903  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:19:52.634696  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:52.788599  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:52.850322  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:52.885003  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:52.887718  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:52.918730  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:52.939492  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:52.945641  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:52.983283  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:52.998788  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:53.023017  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:53.041900  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:53.048334  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:19:53.052302  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:53.098296  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:53.128767  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:53.137315  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:53.139889  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:53.187721  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:19:53.254773  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:53.284287  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:53.291576  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:53.298192  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:53.313794  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:53.317890  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:53.318308  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:53.326256  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:53.419785  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:53.503870  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:53.677420  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:53.705430  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:53.709203  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:53.716529  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:53.741509  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:53.751473  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:53.765264  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:53.782835  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 52693.4us
I20240503 02:19:53.789194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:53.796737  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:53.821893  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:53.826027  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:53.857090  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:53.862164  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:53.903203  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:53.912139  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:53.953907  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:54.194516  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:54.227792  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:54.231666  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:54.267669  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:54.279527  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:54.280829  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:54.337883  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:54.358083  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:54.379173  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:54.390295  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:54.423441  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:54.433104  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:54.466418  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:54.495046  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:54.509522  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:54.515769  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:54.530287  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:54.546336  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:19:54.677445  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:54.737740  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:54.753312  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:54.756155  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:54.761701  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:54.770490  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:54.787362  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:54.810817  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:54.902964  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:19:54.985734  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:19:54.987072  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:19:54.987821  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:54.992568  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240503 02:19:55.147987  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:55.163710  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:55.169610  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:55.180946  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:55.201254  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:55.219117  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:55.268498  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:55.316545  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:55.350203  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:55.377382  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:55.381317  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:55.416081  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:55.439916  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:55.482846  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:55.502368  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:55.532055  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:55.655669  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:55.665748  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:55.670248  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:55.706827  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:55.715608  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:55.744174  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:55.748394  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:55.805595  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:19:55.825573  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:19:55.903270  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:19:56.001818  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:56.054404  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:56.056341  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:56.071043  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:56.117950  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:56.146600  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:56.161274  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:56.189019  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:56.252163  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:56.314445  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:56.344231  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:56.396546  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:19:56.406770  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:56.412781  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:56.442790  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:56.470785  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:56.476910  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:56.483857  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:56.503067  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:56.649590  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:19:56.649823  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:19:56.650884  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:56.686367  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:56.697451  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:56.700627  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:56.728166  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:56.729144  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:56.777639  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:56.791018  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:56.867781  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:57.042488  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:19:57.070304  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:57.076210  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:57.110316  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:57.120898  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:57.150699  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:57.155386  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:57.155361  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:57.176703  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:57.246531  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:57.276098  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:57.286191  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:57.304260  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:57.312384  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:57.347599  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:57.391502  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:57.418884  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:57.607898  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:19:57.623410  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:19:57.625260  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240503 02:19:57.661847  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:57.708720  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:57.750701  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:57.753055  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:57.778151  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:57.781050  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:57.781502  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:57.783365  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:57.820613  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:57.859258  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:57.878521  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:57.880148  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:57.977841  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:57.989066  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:57.997242  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:58.010919  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:58.079030  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:19:58.274181  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:58.293838  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:58.296461  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:58.299844  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:58.300832  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:19:58.319669  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:19:58.336767  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:58.343016  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:58.364403  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:19:58.416623  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:58.627354  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:19:58.686187  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:58.785405  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:58.790624  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:58.802768  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 80227.8us
I20240503 02:19:58.812866  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:19:58.847726  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:19:58.922399  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:58.924729  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:58.950457  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:19:58.953977  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:58.974295  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:58.994417  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:58.997467  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:19:59.011793  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:19:59.012866  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 92414.5us
I20240503 02:19:59.017381  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:59.063299  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:59.100910  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:59.331509  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:59.336180  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:19:59.339288  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:19:59.363883  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:59.371409  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:59.379263  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:19:59.390775  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:19:59.412732  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:59.459273  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:59.608018  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:19:59.611233  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:19:59.644174  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:19:59.663197  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:19:59.667465  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:19:59.670373  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:19:59.677456  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:19:59.694101  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:19:59.749454  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:19:59.755759  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:19:59.774427  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 49416.9us
I20240503 02:20:00.018249  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:00.030629  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:00.044057  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:00.056125  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:00.071018  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240503 02:20:00.073412  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:00.080693  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:00.144731  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:20:00.148295  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:20:00.149027  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:00.169143  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:00.171237  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:00.171370  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:20:00.312891  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:00.336922  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:00.356976  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:00.362191  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:00.363807  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:00.402001  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:00.458463  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:00.641367  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:00.645960  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:00.678144  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:00.687217  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:00.704749  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:00.751231  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:00.786401  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:00.787449  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:00.811599  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:20:01.005890  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:20:01.018725  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:01.026985  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:01.028106  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:01.054486  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:01.065654  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:01.090915  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:01.114476  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:01.142370  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:01.157089  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:01.203647  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:01.328614  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:01.337999  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:01.339082  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:01.349956  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:01.370587  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:01.379025  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:01.402400  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:20:01.413918  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:01.480837  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:01.533129  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:01.591662  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:01.615702  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:01.631065  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:01.646013  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:01.648258  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:01.657743  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:01.671166  2608 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 52971.3us
W20240503 02:20:01.699314  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:01.701447  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:01.886595  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:01.920221  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:01.947026  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:01.954049  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:01.967043  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:01.986032  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:02.024709  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:02.134346  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:20:02.137143  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:02.167114  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:20:02.241654  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:02.300199  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:02.301687  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:02.325394  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:02.326701  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:02.347997  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:02.356009  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:02.359009  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:02.374229  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:02.476058  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240503 02:20:02.484851  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:02.502681  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:02.513962  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:02.535873  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:02.593803  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:02.597009  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:02.599555  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:02.622018  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:02.763330  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:20:02.808724  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:02.850950  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:02.872748  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:02.876879  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:02.889225  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:02.932654  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:02.941736  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:02.973577  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:02.985181  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:02.988881  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:03.164182  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:03.198207  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:03.208959  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:03.210057  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:03.215116  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:03.215484  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:03.245115  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:03.245550  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:03.262887  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:03.445741  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:03.462044  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:03.483286  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:03.497794  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:03.505872  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:03.527426  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:03.559734  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:03.605198  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:03.623517  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:03.694284  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:20:03.871599  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:03.993583  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:04.011122  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:04.014218  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:04.024735  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:04.037204  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:04.056627  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:04.087165  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:04.095166  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:04.276527  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:04.280503  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:04.293390  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:04.293784  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:04.293777  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:04.350781  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:04.362649  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:04.388803  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:04.408989  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 75228.6us
W20240503 02:20:04.435683  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:04.687427  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:04.735702  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:04.738835  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:20:04.792624  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:04.827328  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:04.896016  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:04.902176  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240503 02:20:04.913653  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:05.041788  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:05.042086  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:05.058805  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:05.061743  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:05.082603  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:05.090323  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:05.124574  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:05.130268  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:05.156777  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:20:05.166019  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:05.177207  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:05.198166  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:05.212591  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:05.231807  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:05.235328  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:05.236052  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:05.263470  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:05.303099  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:05.369661  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:05.393678  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:05.394012  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:20:05.437397  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:05.483964  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:05.835728  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:05.902997  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:05.962489  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:05.966445  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:05.985728  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:06.050169  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:06.066354  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:06.068202  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:06.118507  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:06.129690  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:06.172065  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:20:06.204792  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:06.332587  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:06.354768  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:06.401516  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:06.415764  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:06.420425  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:06.423499  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:06.466470  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:06.478760  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:06.594029  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:06.594746  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:06.619304  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:06.642357  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:06.652281  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:06.679056  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:20:06.690151  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:06.716346  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:06.723996  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:06.779882  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:06.827939  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:07.045679  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:07.082420  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:07.083519  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:07.118130  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:07.127745  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:07.127655  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:07.138936  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:07.201589  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:07.220836  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:07.241210  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:07.246791  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240503 02:20:07.272927  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:07.299933  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:07.328008  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:07.368564  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:07.376277  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:07.378607  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:07.379385  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:07.487758  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:07.535849  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:07.549654  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:07.580353  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:07.583439  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:07.646901  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:07.648569  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:07.700362  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:20:07.708673  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:07.734550  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:07.827669  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:07.960638  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:07.992252  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:08.027350  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:08.039808  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:08.050662  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:08.061344  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:08.065474  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:08.071431  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:08.146337  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:20:08.156312  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 59867.5us
I20240503 02:20:08.273929  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:08.282012  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:08.292311  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:08.323154  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:08.372694  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:08.390326  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:08.394052  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:08.424026  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:08.445585  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:08.476426  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:08.663924  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:08.668520  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:08.677726  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:08.707022  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:08.708756  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:08.728662  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:08.731099  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:08.732110  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:08.867645  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:08.922869  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:08.955216  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:08.968849  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:08.991163  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:20:09.066202  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:09.081931  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:09.084961  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:09.139770  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:09.151000  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:09.168758  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:09.309830  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:09.311635  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:09.346921  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:09.360277  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:09.381850  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:09.384909  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:09.389647  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:09.459517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:09.577491  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:09.639961  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:09.643293  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:09.650004  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:09.698091  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:09.698781  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:09.708058  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:09.708703  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:09.729053  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240503 02:20:09.730234  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:09.853925  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:09.866523  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:09.987152  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:09.996703  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:10.021615  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:10.022985  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:10.028810  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:10.052820  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:10.096220  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:10.111196  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:10.263777  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:10.265311  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:10.267343  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:10.285877  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:10.287258  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:10.326563  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:10.337829  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:10.346663  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 47290us
W20240503 02:20:10.346976  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:10.396603  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:10.581032  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:10.601812  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:10.619623  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:10.620114  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:10.711082  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:10.723521  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:10.725602  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:10.731366  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 44679.8us
I20240503 02:20:10.732751  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:10.823810  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:10.854822  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:10.868923  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:10.893478  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:10.909586  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:10.921774  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:10.946758  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:10.986117  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:10.997395  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:11.074689  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:11.115439  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:11.231642  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:11.238377  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:11.241000  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:11.250272  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:11.259740  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:11.271416  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:11.278565  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:11.329428  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:20:11.350418  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:20:11.356940  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:20:11.438989  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:11.595995  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:11.620440  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:11.660969  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:11.684595  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:11.685907  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:11.686168  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:11.721087  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:11.754740  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:20:11.761453  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:11.766109  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:11.774694  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:11.779628  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:11.845592  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:11.862144  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:11.864871  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:11.885725  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:11.909665  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:11.952669  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:12.041970  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:12.150770  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:12.165750  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:12.169751  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:12.172644  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:12.181836  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:12.192514  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:12.249467  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240503 02:20:12.257159  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:12.331297  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:20:12.401765  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:12.425554  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:12.486410  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:12.503554  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:12.512089  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:12.515552  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:12.549008  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:12.555253  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:12.599731  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:12.763192  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:12.801967  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:12.810329  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:12.870565  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:12.881034  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:12.902782  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:20:12.933424  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 77349.4us
I20240503 02:20:12.972416  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:12.994282  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:12.994475  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:13.059759  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:20:13.186137  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:13.204859  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:13.223001  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:13.223634  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:13.225762  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:13.264531  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:13.296979  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:13.308324  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:13.400187  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:20:13.401363  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:13.439648  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:13.445852  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:13.462749  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:13.482726  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:13.489600  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:13.501587  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:13.545214  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:13.585884  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:13.759866  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:13.807166  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:13.815150  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:13.862641  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:20:13.884451  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:13.889995  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:13.922809  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:13.952216  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:13.973469  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:14.035089  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:14.049907  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:14.060161  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:14.098650  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:14.118098  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:14.141414  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:14.148196  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:14.166322  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:14.190500  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:14.215875  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:14.262691  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:14.342600  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:14.350343  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:14.400007  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:14.407459  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:14.424826  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:14.447744  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:14.489270  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:14.494089  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:14.638674  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240503 02:20:14.733049  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:14.737071  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:14.747094  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:14.750567  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:14.755414  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:14.757546  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:14.794224  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:14.805658  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:14.815958  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:14.840204  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:14.936887  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:15.048813  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:15.049013  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:15.089995  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:15.092386  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:15.120707  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:15.135804  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:15.165295  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:15.180104  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:15.221817  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:15.313316  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:15.315007  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:15.327481  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:15.341850  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:15.345880  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:15.374588  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:15.398334  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:15.401253  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:15.609982  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:15.631985  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:15.652042  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:15.669453  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:15.671859  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:15.712224  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:15.721282  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:15.727501  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:15.750218  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:15.750665  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:15.755503  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:15.788396  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:15.805917  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:15.814322  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:15.834327  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:15.857136  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:15.861642  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:20:16.001312  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 91290.9us
W20240503 02:20:16.006497  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:16.149559  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:16.156781  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:16.158177  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:16.172609  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:16.174469  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:16.180001  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:16.184578  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:16.205632  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:16.223841  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:20:16.381495  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:20:16.406700  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:16.507972  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:16.515467  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:16.539417  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:16.540611  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:16.542452  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:16.592537  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:16.639418  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:16.658401  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:16.663887  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:16.683081  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:16.701649  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:16.724795  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:16.758785  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:16.766577  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:16.800426  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:16.840672  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:16.967341  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:20:16.999644  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:20:17.010818  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:20:17.103801  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240503 02:20:17.163684  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:17.165511  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:17.175823  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:17.213297  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:17.238689  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:17.243253  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:17.248886  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:17.251780  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:17.261168  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:17.262116  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:17.459177  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:17.489012  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:17.503871  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:17.503952  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:17.547952  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:17.582103  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:17.617731  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:17.630190  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:17.693663  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:17.795490  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:17.822010  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:17.867691  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:17.903381  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:17.945168  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:17.965404  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:17.976332  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:18.014334  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 87308.2us
W20240503 02:20:18.114121  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:20:18.129371  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:18.197988  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:18.213104  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:18.218089  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:18.232173  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:18.276929  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:18.278120  2478 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 46249.4us
I20240503 02:20:18.296231  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:18.303699  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:18.340035  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:18.353209  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:20:18.385757  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 131093us
W20240503 02:20:18.489398  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:20:18.523880  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:18.588093  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:18.628110  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:18.631042  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:18.669898  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:18.674966  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:18.698436  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:18.720376  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240503 02:20:18.746367  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:18.916882  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.024500  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:19.033510  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.066861  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:19.076601  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:19.088299  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:19.100194  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:19.102015  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 95111.6us
I20240503 02:20:19.113816  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:19.117692  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:19.129217  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:19.147344  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:19.157567  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 69263.7us
W20240503 02:20:19.241816  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:20:19.339219  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.342339  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:19.366328  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:19.385885  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:19.387028  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:19.410038  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:19.421002  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:19.422055  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:19.436725  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.439713  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:19.498346  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:19.537117  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:19.541086  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:19.543162  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:19.574047  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:19.577232  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:19.658499  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.675148  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:19.792136  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240503 02:20:19.881749  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:19.904776  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:19.920589  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:19.939221  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:20.014142  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:20.044555  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:20.052697  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:20.152534  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:20:20.157351  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:20.342541  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:20:20.349905  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:20.353060  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:20.354051  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:20.354846  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:20.385423  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:20.392370  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:20.442659  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:20.552507  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:20.590427  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:20.605726  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:20.618395  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:20.659000  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:20.659183  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:20.698240  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:20.705781  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:20.726831  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:20:20.829609  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:20.988549  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:21.010279  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:21.016521  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:21.059581  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:21.116627  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:21.121155  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:21.155555  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:21.156862  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:21.177929  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:21.262272  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:21.278218  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:21.303239  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:21.306453  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:21.332538  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:21.353277  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:21.397984  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:21.444015  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:20:21.453572  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:20:21.534703  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:21.591274  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:21.593005  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:21.611116  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:21.611830  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:21.613586  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:21.621608  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:21.629735  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:21.720490  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:21.722270  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:20:21.771700  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:20:21.995645  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:22.056633  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:22.071349  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:22.072448  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:22.087445  2520 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:22.087776  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:22.088227  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:22.122745  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:22.156093  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:22.202939  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:22.212898  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:22.222929  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:22.242127  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:20:22.247386  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:22.249388  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:22.267263  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:22.285392  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:22.298312  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:22.323037  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:22.350728  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 77671.8us
W20240503 02:20:22.352975  2475 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 126932us
W20240503 02:20:22.366972  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:22.606514  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:22.698243  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:22.707276  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:22.729282  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:22.762884  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:22.764168  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:22.790163  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:22.814733  2606 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 106640us
I20240503 02:20:22.857584  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:22.865728  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:22.882699  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:22.891132  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:22.916194  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:22.918100  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:22.969749  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:22.977834  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:23.027490  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:23.230434  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:23.231101  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:23.245366  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:23.251801  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:23.262645  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:23.263070  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:23.285427  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:23.286427  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:23.351270  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:23.424728  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:23.533751  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:23.606680  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:23.637095  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:23.640070  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:23.671288  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:23.683203  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:23.687757  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:23.718537  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:23.722826  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:23.727644  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:23.768133  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:23.785262  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:23.821076  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:23.844774  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:23.849614  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:23.910097  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:23.912395  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:23.939356  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240503 02:20:23.941921  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:24.180306  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:24.183302  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:24.186888  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:24.216883  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:24.241654  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:24.272455  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:24.280565  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:24.303879  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:24.359118  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:24.425235  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:24.429805  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:24.436687  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:20:24.459844  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:24.476665  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:24.501948  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:24.506582  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:24.516742  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:24.545645  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:24.550365  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:24.704720  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:20:24.729820  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:24.739969  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:24.797108  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:24.802124  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:24.808423  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:24.854508  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:24.863040  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:24.866717  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:24.870353  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:24.913241  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:25.045439  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:25.049701  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 147538us
I20240503 02:20:25.115029  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:25.150630  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:25.253948  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:25.300952  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:25.383687  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:25.411336  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:20:25.434444  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:25.441941  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:25.464761  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:25.586133  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:25.608475  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:20:25.613724  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:25.665556  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:25.688846  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:25.714087  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:25.721807  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:25.726554  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:25.863395  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:26.034047  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:26.152840  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:26.171510  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:26.172202  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:26.181648  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:26.194983  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:26.202872  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:26.212980  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:20:26.248034  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:26.285691  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:26.346630  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:26.368772  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:26.396731  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:26.413966  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:26.431769  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:26.462042  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:20:26.463325  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:26.488281  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:26.508635  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240503 02:20:26.529008  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:26.643364  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:26.643826  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:26.650285  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:26.669801  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:26.673919  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:26.691408  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:26.752242  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:26.850277  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:26.851722  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:20:26.893107  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:27.033718  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:27.042841  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:27.052031  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:27.089704  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:27.107373  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:27.110222  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:27.115159  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:27.143774  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:27.192490  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:27.213114  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:27.292096  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:27.299813  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:27.303567  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:20:27.305167  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:20:27.341557  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:27.353798  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:27.354770  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:27.365501  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:27.401904  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:27.544361  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:27.553207  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:27.627090  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:27.654120  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:27.655223  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:27.666338  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:27.699613  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:27.711205  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:27.800942  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:28.045161  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:28.102316  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:28.167984  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:28.243914  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:28.255010  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:28.257158  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:28.284502  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:28.287267  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:28.301448  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:28.507253  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:28.533010  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:28.569506  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:28.570459  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:28.574497  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:28.575435  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:28.581856  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:28.585212  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:28.606586  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:28.607558  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:28.607802  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:28.629989  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:28.672719  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:28.686201  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:28.726610  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:28.729481  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:28.754202  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:28.894376  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:20:28.894739  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:29.042105  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:29.053469  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:29.059095  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:29.067801  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:29.086939  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:29.089699  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:29.099421  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:29.230118  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:29.308702  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:20:29.377290  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240503 02:20:29.379792  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:29.391858  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:29.410679  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:29.415746  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:29.418123  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:29.420055  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:29.422940  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:29.463354  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:29.473169  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:29.476290  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:29.507674  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:29.508309  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:29.528789  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:29.554013  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:29.570358  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:29.614269  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:29.632077  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:29.642055  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240503 02:20:29.651834  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:29.670920  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:29.822768  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:29.874505  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:29.882686  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:29.895242  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:29.915309  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:29.926914  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:29.970176  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:29.973215  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 47760.6us
I20240503 02:20:29.983538  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:30.080273  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:30.263366  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:30.274963  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:30.276095  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:30.319159  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:30.343329  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:30.354990  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:30.399008  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:30.399848  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:30.581681  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:30.582584  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:30.596486  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:30.603139  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:30.620003  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:30.650007  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:30.663275  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:30.710337  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:20:30.712466  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:30.759416  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:30.914861  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:30.925695  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:30.926718  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:30.937292  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:30.948890  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:30.977767  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:31.009033  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:31.096961  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:31.178545  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:31.201056  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:31.210705  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:31.213460  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:31.297818  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:31.309073  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:31.323609  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:31.333745  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:31.364086  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:31.407637  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:31.540452  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:31.581665  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:31.597781  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:31.618851  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:31.640645  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:31.650203  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:31.668094  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:31.701933  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:20:31.759392  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:20:31.760602  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:31.799430  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:31.878188  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:31.885429  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:32.012360  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:32.016309  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:32.041394  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:32.044479  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:32.085547  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:32.091358  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:32.097316  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:32.110900  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:32.111646  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:32.142094  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:32.198194  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:32.215427  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:32.228708  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240503 02:20:32.243955  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:32.250890  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:32.274904  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:32.282917  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:32.296553  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:32.480906  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:32.495872  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:32.512490  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:32.525763  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:32.532483  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240503 02:20:32.533560  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:32.538122  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:32.574262  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:32.575794  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:32.632200  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:32.862197  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:32.881004  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:32.933812  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:32.967533  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:32.969830  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:32.969825  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:32.972252  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:33.006109  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:33.011516  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:33.069805  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:33.074402  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:33.078173  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:33.086163  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:33.090612  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:33.095566  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:33.147224  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:33.199882  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:33.363204  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:33.387038  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:33.420361  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:33.421434  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:33.421958  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:33.481984  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:33.485059  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:33.651543  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:33.660872  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:33.675489  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:33.698366  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:33.698868  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:33.705376  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:33.724999  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:33.769974  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:33.771263  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:33.949620  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:34.002710  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:34.048605  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:34.056223  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:34.075644  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:34.078609  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:34.102777  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:34.135746  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:34.160638  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:34.199216  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:34.257200  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240503 02:20:34.365882  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:34.382273  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:34.382555  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:34.385412  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:34.386582  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:34.391948  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:34.431679  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:34.434162  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:34.456007  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:34.545008  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:20:34.608124  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:34.649775  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:34.667171  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:34.685940  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:34.686913  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:34.697575  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:34.699630  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:34.724146  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:34.754715  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:34.758666  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:20:34.761287  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:34.772656  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:34.883595  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:35.025242  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:35.054816  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:35.067656  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:35.075276  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:35.086957  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:35.145046  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:35.159076  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:35.181984  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:35.371791  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:35.379452  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:35.407023  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:35.410145  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:35.441128  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:35.444512  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:35.450421  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:35.490644  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:35.498082  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:35.556988  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:35.563319  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:35.587905  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:35.618501  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:35.633944  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:35.651232  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:35.686941  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:20:35.765442  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:35.784725  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:35.916993  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:35.925204  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:35.932706  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:35.933704  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:35.962559  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:35.993033  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:36.011327  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:36.056542  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:20:36.249914  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:36.300419  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:36.314908  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:36.321425  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:36.323951  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:36.332042  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:36.340770  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:36.375257  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:36.384902  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:36.386812  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:36.391408  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:36.424619  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:36.496155  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:36.502493  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:36.525689  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:36.543179  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:36.673171  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:20:36.702524  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:20:36.703569  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:36.735519  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:36.825388  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:36.872771  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:36.927667  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 118276us
W20240503 02:20:36.931242  2217 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 97295.7us
I20240503 02:20:36.946481  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:36.950532  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:36.978678  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:37.006305  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:37.043604  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240503 02:20:37.048521  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:37.126348  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:37.147955  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:37.196453  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:37.197464  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:37.217537  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:37.222934  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:37.263151  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:37.295125  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:37.306634  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:37.316546  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:20:37.326623  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:20:37.450589  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:37.475418  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:37.484913  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:37.488142  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:37.508432  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:37.523195  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:37.534229  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:37.568218  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:37.568974  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:37.630765  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240503 02:20:37.781689  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:37.783743  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:37.808760  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:37.829839  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:37.834530  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:37.875087  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:37.925109  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:37.932016  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:37.961081  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 62065.6us
I20240503 02:20:38.170670  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:38.171723  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:38.198272  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:38.199476  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:38.218746  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:38.238113  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:38.304363  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:38.329473  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:38.348436  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:38.387740  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:20:38.460604  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:38.473954  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:38.498485  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:38.547160  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:38.577966  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:38.633708  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:38.652218  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:38.665876  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:38.721419  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:38.789120  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:38.964839  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:38.965821  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:38.990500  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:39.001639  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:39.049041  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:39.085916  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:39.086341  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:39.119740  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:39.123167  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:39.135571  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:39.157868  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:39.164649  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:39.179414  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240503 02:20:39.179948  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:39.191278  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:39.314968  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:39.393074  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:39.415333  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:39.417610  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:39.419404  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:39.420128  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:39.425642  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:39.443782  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:39.447604  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:39.463943  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:39.605741  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:20:39.679831  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:20:39.801707  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:39.854352  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:20:39.917958  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 53008.3us
W20240503 02:20:39.919767  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:39.950137  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:40.044428  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:40.045897  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:40.050781  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:40.059937  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:40.068791  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:40.071960  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:40.094028  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:20:40.104092  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:40.150137  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:40.165257  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:40.170460  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:40.174432  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:40.178788  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 84750.2us
W20240503 02:20:40.182866  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:40.237967  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:40.254061  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:40.257238  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:40.313689  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:40.506350  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:40.524070  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:40.539819  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:40.571202  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:40.577143  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:40.597878  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:40.657905  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:40.790632  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:20:40.869411  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:40.944778  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:40.968919  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:40.985674  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240503 02:20:41.001521  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:41.021260  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:41.070848  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:41.071435  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:41.098537  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:41.108382  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:41.143909  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:41.146620  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:41.157096  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:41.189715  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:41.204216  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:41.219117  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:41.227566  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:41.266489  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:41.294986  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:41.472975  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:41.475553  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240503 02:20:41.482704  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:41.497961  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:41.502938  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:41.506094  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:41.555727  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:41.556952  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:41.560684  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:41.814832  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:20:41.816609  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:41.876370  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:41.886857  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:41.901002  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:41.919209  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:41.923054  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:41.967737  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:41.990713  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:42.013509  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:20:42.016913  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:20:42.053810  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:42.059018  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:42.074329  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:42.091208  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:42.116483  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:42.137913  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:42.203867  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:42.224040  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:42.251443  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:42.256448  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240503 02:20:42.391252  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:42.399981  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:42.422956  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:42.423139  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:42.424194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:42.505868  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:42.511152  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:42.536573  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:20:42.673743  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:42.717623  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:42.730554  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:42.746071  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 32 similar messages]
I20240503 02:20:42.751227  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:42.751511  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:42.765352  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:42.817590  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:42.836676  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:42.842854  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:43.048964  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:43.072438  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:43.098575  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:43.129184  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:43.130939  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:43.145993  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:43.166363  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:43.234325  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:43.259616  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:43.369482  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:43.409698  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:43.417745  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:43.437790  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:43.443224  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:43.449203  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:43.460016  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:43.558447  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:43.560078  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:43.656582  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:20:43.658226  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:43.731716  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:43.765729  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:43.786151  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:43.796494  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:43.797205  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:43.826047  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:43.829041  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:43.928517  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:44.058995  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:20:44.079301  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:44.092603  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:44.127302  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:44.163384  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:44.194518  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:44.240963  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:44.265152  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:44.285075  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:44.403725  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:44.433864  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:20:44.445709  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:44.521806  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:44.540500  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:44.541646  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:44.561633  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:44.598886  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:44.602173  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:44.605957  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:44.659433  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:44.684804  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:44.719570  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:44.739481  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:44.742594  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:44.758481  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 47916.5us
W20240503 02:20:44.767014  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:20:44.770998  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:44.782233  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:44.783461  4260 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:34903, user_credentials={real_user=slave}} blocked reactor thread for 65095.3us
I20240503 02:20:44.820317  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:44.826967  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:20:44.919939  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:45.142856  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:45.143898  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:45.155203  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:45.174937  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:45.178903  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:45.202009  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:45.218225  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:45.219764  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:45.388641  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:20:45.389750  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:20:45.449899  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:45.500634  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:45.515380  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:45.603543  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:45.611428  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:45.630303  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:45.650511  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:45.675208  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:45.726794  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:45.798117  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:45.815152  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:45.845055  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:45.854341  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:45.867590  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:45.890465  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:45.900746  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:45.928726  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:45.930904  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:45.956570  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:46.144984  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:46.171247  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:46.171387  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:46.171717  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:46.182361  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:46.186331  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:46.202124  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:46.204834  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:46.217155  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:46.274085  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 107305us
W20240503 02:20:46.277019  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:46.452545  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:46.480224  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:46.515112  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:46.518177  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:46.565044  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:46.589205  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:46.590099  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:46.590874  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:46.624692  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:46.680160  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:46.786751  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:46.806181  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:46.846674  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:46.850196  2345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 55605.2us
I20240503 02:20:46.852566  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:46.876391  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:46.880057  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:46.898236  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:46.913771  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:20:46.915776  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240503 02:20:46.939613  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:47.154348  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:47.154479  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:47.183984  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:47.187501  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:47.188462  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:47.203853  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:47.227145  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:47.251780  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:47.385588  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:47.427824  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:47.428969  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:47.439110  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:47.445178  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:47.457813  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:20:47.459434  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240503 02:20:47.467087  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:47.540055  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:47.567265  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:47.708293  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:47.731607  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:47.734633  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:47.767127  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:47.783955  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:20:47.835139  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:47.887883  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:47.893311  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:47.898041  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:48.004997  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:48.029886  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:20:48.124287  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:48.134299  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:48.154371  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:48.165889  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:48.167968  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:48.179683  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:48.191072  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:48.215473  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:48.239672  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:48.250443  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:48.254561  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:48.283289  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:48.286140  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:48.290062  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:48.314628  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:48.355010  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:48.376987  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:48.648823  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:20:48.681428  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:48.706785  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:48.714994  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:48.722402  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:48.749119  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:48.772843  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:48.777176  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:48.805320  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:49.001183  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:20:49.051589  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:49.134802  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:49.147140  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:49.161363  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:49.162775  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:49.228446  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:49.254639  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:49.303447  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:49.312119  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:49.318410  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:49.347208  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:49.359647  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:49.360628  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:49.372330  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:49.380728  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:49.413884  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:49.434441  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:49.654515  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:20:49.681716  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:49.692960  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:49.696760  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:49.718533  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:49.733207  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:49.750267  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:49.755097  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:49.820677  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:49.951798  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:20:49.973552  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:20:49.985955  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:50.041410  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:50.045518  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:50.047482  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:50.074949  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:50.078100  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:50.095348  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:50.128031  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:50.167183  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:50.167876  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:50.349666  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:50.363096  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:50.377499  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:50.399859  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:50.418231  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:50.448700  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:50.454725  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:50.498656  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:50.547295  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:20:50.580976  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:50.582347  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:20:50.614560  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:50.617703  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:50.695792  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:50.722295  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:50.757555  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:50.766976  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:50.785051  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:50.850175  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:50.913762  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:51.061055  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:51.085875  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:51.086740  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:51.111750  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:51.126539  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:51.130854  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:51.146762  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:51.164357  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:51.236322  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:51.335518  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:51.356120  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:51.367655  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:51.371491  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:51.379817  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:51.407135  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:51.422648  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240503 02:20:51.430485  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:51.442335  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:20:51.521087  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:51.744679  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:51.746817  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:51.758695  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:51.782229  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:51.786232  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:51.798837  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:51.834918  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:51.846429  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:51.847445  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:51.897141  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:51.901360  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:51.981194  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:51.999233  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:20:52.000581  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:52.003284  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:52.009548  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:52.024642  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:52.028988  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:52.240062  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:52.261152  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:52.263329  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:52.302222  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:52.304752  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:52.307518  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240503 02:20:52.322496  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:52.353451  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:52.374172  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:52.392071  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:52.462045  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:52.467327  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:52.650007  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:52.664211  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:52.694689  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:52.710407  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:52.716226  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:52.719878  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:52.721918  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:52.845501  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:20:52.888381  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:52.912271  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:52.953891  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:52.994383  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:53.015302  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:53.022354  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:53.028810  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:53.036657  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:53.054468  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:53.068672  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:53.075768  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:53.211094  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:53.236382  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:53.237505  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:53.253160  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:53.269068  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:53.284432  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:53.287266  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:53.359752  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:53.504995  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:53.532086  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:53.545048  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:53.546288  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:53.585357  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:53.594166  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:53.609947  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:53.640138  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:53.655086  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:53.719748  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 43757us
I20240503 02:20:53.803187  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:53.837872  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:53.847402  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:53.888939  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:53.929759  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:53.947006  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:53.953025  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:53.987569  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:20:54.163756  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:54.165503  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:54.166592  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:54.179476  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:54.189532  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:54.190671  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:54.208591  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:54.227691  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:54.239260  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:20:54.397117  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:54.409897  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:54.418677  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:54.419217  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:54.433689  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:54.463868  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:54.513447  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:54.514385  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:54.757292  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:54.791934  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:54.792534  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:54.824533  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:54.846210  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:54.869467  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240503 02:20:54.892010  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:54.914966  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:54.926096  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:54.943207  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:54.964752  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:55.043259  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:55.047318  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:55.064220  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:55.064618  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:55.111696  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:55.114207  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:55.130946  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:55.223686  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:55.265913  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:55.266258  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:55.275713  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:20:55.286407  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:55.288304  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:55.289889  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240503 02:20:55.299680  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:55.311286  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:55.320722  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:55.465284  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:55.547091  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:55.562626  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:55.564862  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:55.570914  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:55.607811  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:55.614720  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:55.649226  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:55.662528  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:20:55.692236  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:55.698509  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:20:55.723248  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:20:55.958263  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:55.984423  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:56.003432  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:56.039798  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:56.039995  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:56.075069  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:56.080726  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:56.090051  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:56.237349  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:56.266784  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:56.293540  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:56.300208  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:56.326850  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:56.366721  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:56.422858  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:56.482421  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:56.519188  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:56.546607  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:56.553292  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:56.572808  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:20:56.576989  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:56.579854  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:56.642376  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:56.661024  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:56.683115  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:56.749789  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:20:56.782306  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:57.018314  2260 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:57.018498  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:20:57.029168  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240503 02:20:57.030215  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:57.041214  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:57.055492  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:20:57.068429  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:57.082940  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:57.112761  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:57.130348  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:57.189886  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:57.215572  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:57.252800  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:57.284579  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240503 02:20:57.284832  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:57.285593  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:57.312116  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:57.327693  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:57.367835  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:57.455503  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:20:57.565624  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240503 02:20:57.620357  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:57.631683  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:57.638821  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:57.639611  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:57.644212  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:57.667547  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:57.701414  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:57.749480  2477 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 80590.5us
W20240503 02:20:57.750365  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 110121us
W20240503 02:20:57.766693  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:57.853320  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240503 02:20:57.907418  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:20:58.048887  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:20:58.085565  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:58.102373  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:58.118500  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:58.121057  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:58.142483  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:58.168828  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:58.202545  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:58.222240  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:20:58.245438  2214 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.70:33893, user_credentials={real_user=slave}} blocked reactor thread for 67000.7us
I20240503 02:20:58.296144  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:58.327838  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:58.331135  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:58.337178  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:20:58.354895  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:20:58.417548  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:58.431831  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:20:58.446689  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:58.489094  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:58.610881  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:20:58.726800  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:58.754293  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:58.777705  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:58.790155  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:58.809967  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:58.875448  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:58.898166  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:58.902634  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:58.940501  2609 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 110448us
W20240503 02:20:58.941994  4131 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 106573us
W20240503 02:20:58.966022  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:20:59.116114  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:59.117169  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:59.143399  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:59.170228  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:59.172178  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:59.208928  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:59.209442  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:20:59.218698  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:59.218811  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:59.262842  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:59.302979  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:20:59.303736  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:59.320932  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:20:59.338582  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 48006.3us
I20240503 02:20:59.338737  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:59.344429  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:20:59.505591  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:20:59.556318  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:20:59.568993  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:20:59.574991  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:20:59.598569  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:20:59.601428  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:20:59.601847  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:20:59.608229  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:20:59.622629  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:20:59.623544  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:20:59.843565  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:21:00.014024  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:00.049402  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:00.064360  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 02:21:00.071381  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:21:00.073720  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:21:00.073931  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:00.094991  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:00.134002  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:21:00.134979  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:00.137475  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:00.140108  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:00.223017  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:00.256335  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:00.259119  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:00.265851  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:00.297724  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:00.329550  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:00.331142  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:00.440246  2346 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 44856.5us
W20240503 02:21:00.489208  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:21:00.491595  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:21:00.547039  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:00.601593  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:00.605034  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:00.639374  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:00.676591  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:00.693897  4262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240503 02:21:00.699879  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:00.733775  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:21:00.759949  2216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:21:00.761608  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:21:00.867005  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:21:00.969086  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:00.974692  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:00.994357  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:01.011299  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:01.026142  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:21:01.034832  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:01.047448  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:01.070798  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:01.090869  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:01.209398  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:01.222028  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:01.258757  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:01.261196  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:01.301692  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:01.331945  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:01.358284  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:01.391670  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:21:01.495079  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:01.556811  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:01.558892  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:01.579795  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:01.619437  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:01.704617  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:01.733156  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:01.736635  2608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240503 02:21:01.754236  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:21:01.913604  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:21:01.970013  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:21:02.018867  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:02.030870  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:21:02.057524  2477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240503 02:21:02.061256  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:02.091430  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:02.106539  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:21:02.113457  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:21:02.126647  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:02.138152  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:02.201020  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:21:02.227313  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:21:02.351716  4262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 48073us
W20240503 02:21:02.365687  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:02.384559  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:02.392590  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:02.415823  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:02.431232  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:02.543609  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:02.552158  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:02.563038  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:21:02.577616  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 83841.3us
W20240503 02:21:02.635645  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:21:02.640342  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:02.687124  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:02.699750  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:02.720271  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:02.726250  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:02.738528  2608 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:21:02.739197  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:02.752498  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:02.769105  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240503 02:21:02.776003  4262 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240503 02:21:02.784423  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:02.803725  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:02.807212  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:02.808408  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:02.830041  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:21:02.863328  2347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240503 02:21:02.923008  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:02.928804  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:03.066004  2347 consensus_peers.cc:485] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:21:03.097800  2608 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:03.193445  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:03.206045  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:03.222433  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:03.228472  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:03.244489  2216 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:03.251128  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:03.253904  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:03.258697  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:03.287972  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:21:03.568936  2477 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:21:03.597909  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:21:03.608942  2347 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 49950.1us
I20240503 02:21:03.626999  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:03.631326  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:03.633193  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:03.644230  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:03.657156  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:03.659135  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:03.662647  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:03.685268  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:03.700462  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:03.747400  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:03.756392  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:03.805569  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:03.807169  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:03.822423  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:04.019731  2608 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:04.105487  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:04.133318  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:04.137981  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:04.158659  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:04.173601  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:04.190289  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:04.193003  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:04.227524  2347 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:21:04.310456  2347 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:04.312228  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:04.320724  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240503 02:21:04.330312  2477 consensus_peers.cc:462] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:04.353771  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:04.396044  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:04.399744  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:04.430790  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:04.446178  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:04.521492  2608 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:04.586443  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:04.592626  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:04.607137  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:04.616485  2477 consensus_peers.cc:485] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): Couldn't send request to peer 7fd07044862a481aa387698017467c01. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:21:04.626324  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:04.666374  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:04.730696  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:04.741559  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:04.752277 18562 raft_consensus.cc:998] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.344s)
I20240503 02:21:04.753016 18562 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:04.765115  4326 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:04.765888 18641 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:04.771369 18641 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:04.780082  4326 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:04.786792  2133 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
W20240503 02:21:04.833187  2126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:04.855670 18710 raft_consensus.cc:998] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.028s)
I20240503 02:21:04.858862 18710 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
W20240503 02:21:04.859280  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51dbf0fd54b14c2fa21d3e62104c9b87: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:04.862469  2540 consensus_queue.cc:237] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:04.866904  4326 raft_consensus.cc:1261] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240503 02:21:04.868147  2475 consensus_peers.cc:485] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:04.868638 18641 consensus_queue.cc:1035] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:04.871896 18562 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:04.872572  4326 raft_consensus.cc:2932] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:04.877836  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:04.880834  2133 catalog_manager.cc:5484] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 8 to 9, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:04.901738  4461 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:04.902647 18712 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:21:04.922022  2216 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 64456.8us
I20240503 02:21:04.931818 18712 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:04.937650  4460 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:04.945837  2133 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:04.962538  2281 consensus_queue.cc:237] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
W20240503 02:21:04.974130  2215 consensus_peers.cc:485] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e0b954008c4f30af414da6cb5d59c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:04.977691  4461 raft_consensus.cc:1261] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08055479ea104aa290d19955d3720850: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:04.979650 18723 consensus_queue.cc:1035] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:04.994761 18723 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.001767  4461 raft_consensus.cc:2932] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.006901  2127 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 02e0b954008c4f30af414da6cb5d59c2 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.010349  2133 catalog_manager.cc:5484] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New cstate: current_term: 1 leader_uuid: "08055479ea104aa290d19955d3720850" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.042230 18416 raft_consensus.cc:998] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.299s)
I20240503 02:21:05.043120 18416 consensus_queue.cc:237] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:05.046821  4461 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.047694 18757 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.048359  2540 raft_consensus.cc:1261] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.049043 18416 consensus_queue.cc:1035] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.054185 18562 raft_consensus.cc:998] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.006s)
I20240503 02:21:05.055186 18562 consensus_queue.cc:237] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:05.071183 18753 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.077288  2540 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.083029  4196 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.083525  2133 catalog_manager.cc:5484] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.083906 18641 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.089388 18705 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.089388  4461 raft_consensus.cc:2932] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.095908  4196 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.103952  2135 catalog_manager.cc:5484] T 913d0aa014ad4bcaac905b5fdaa296d2 P 1d36807868fe4a82bca42c25831ce8a2 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.111747  2669 raft_consensus.cc:1261] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.112929 18641 consensus_queue.cc:1035] T 913d0aa014ad4bcaac905b5fdaa296d2 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.131016 18693 raft_consensus.cc:998] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.079s)
I20240503 02:21:05.131951 18693 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:05.146764  2669 raft_consensus.cc:2932] T 913d0aa014ad4bcaac905b5fdaa296d2 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.172394  4461 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:05.173310 18783 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:05.180382  4461 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.187310  2139 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b reported cstate change: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.190480 18442 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.204736 18785 raft_consensus.cc:998] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.183s)
I20240503 02:21:05.205385 18785 consensus_queue.cc:237] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:21:05.207238  2410 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
W20240503 02:21:05.211222  2346 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.211309  4461 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:05.212095 18442 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.215543 18442 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.218312  4461 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.220934  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8870feaf47a64a929dd2e584ece47218 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.223860  2133 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.225039 18770 raft_consensus.cc:998] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.504s)
I20240503 02:21:05.225744 18770 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:05.229759  4196 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:21:05.230393 18562 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.233016 18770 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.233685  2281 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:05.234617 18785 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:05.236621 18785 raft_consensus.cc:998] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.248s)
I20240503 02:21:05.238090 18785 consensus_queue.cc:237] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:05.241914  4325 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:05.242158  2133 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.242796  4196 raft_consensus.cc:1261] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:05.242713  4195 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.243201 18785 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.243701 18777 consensus_queue.cc:1035] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.244367 18756 raft_consensus.cc:998] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.113s)
I20240503 02:21:05.244928 18756 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:05.246531 18777 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.247673  2281 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.249635  2540 consensus_queue.cc:237] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:21:05.250360 18770 raft_consensus.cc:998] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.384s)
I20240503 02:21:05.250995 18770 consensus_queue.cc:237] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:05.255259  4326 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:05.255856 18753 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:05.258016 18442 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.258641  4196 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.265406  4194 raft_consensus.cc:1261] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:05.266552  2139 catalog_manager.cc:5484] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.272996  2410 consensus_queue.cc:237] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:05.275985  2133 catalog_manager.cc:5484] T 9b822c691d824b2ba25e967535c9734e P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.278425  2281 raft_consensus.cc:1261] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 6d54feb0bbc14c109783371d2939b234: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:21:05.278601  2346 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.279080 18785 consensus_queue.cc:1035] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.283107  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240503 02:21:05.288401  2477 consensus_peers.cc:485] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Couldn't send request to peer e9fdaf326bb64edabc43b366df870f83. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.288736 18641 consensus_queue.cc:1035] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:05.292042 18562 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:21:05.297744  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d2351807a2ed47419f8d674fbeb9d0ec with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.298964  4194 raft_consensus.cc:2932] T d2351807a2ed47419f8d674fbeb9d0ec P 1d36807868fe4a82bca42c25831ce8a2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:21:05.299963 16912 raft_consensus.cc:998] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.197s)
I20240503 02:21:05.300684 16912 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:05.302006 18785 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.303333  2139 catalog_manager.cc:5484] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 9 to 10, VOTER e9fdaf326bb64edabc43b366df870f83 (127.20.218.68) added. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } }
I20240503 02:21:05.310031  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a7ab8b0c8db54565a6066b9a7d1569d8 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.312258  2281 raft_consensus.cc:2932] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.317867  2410 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:21:05.318774 18807 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.323381 18753 raft_consensus.cc:998] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.143s)
I20240503 02:21:05.324043 18753 consensus_queue.cc:237] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:21:05.326943  2133 catalog_manager.cc:5484] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 6 to 7, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New cstate: current_term: 1 leader_uuid: "6d54feb0bbc14c109783371d2939b234" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.355633 16912 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.356591  2410 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.363126  2131 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 2 leader_uuid: "9e699d9f984b4cf2b8b468322759b78c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.370908  2279 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.371263  2281 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:05.371727 18641 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.371994 18756 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:05.373987  4461 raft_consensus.cc:1261] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.374749 18641 consensus_queue.cc:1035] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.378593  4326 raft_consensus.cc:2932] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.380307 18753 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.382369  4325 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.390194 18756 raft_consensus.cc:998] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83: Attempting to remove follower 7fd07044862a481aa387698017467c01 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 7fd07044862a481aa387698017467c01 for more than 300 seconds (300.415s)
I20240503 02:21:05.390887 18756 consensus_queue.cc:237] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:05.394954  4194 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.394979  2540 raft_consensus.cc:1261] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.395583 18756 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.395943 18416 consensus_queue.cc:1035] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.410058 18641 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.412842 18756 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.413372  4194 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.414139 18757 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.419548  2280 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.419871  2133 catalog_manager.cc:5484] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.426112  2131 catalog_manager.cc:5484] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.428450  2133 catalog_manager.cc:5484] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 5 to 6, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } }
I20240503 02:21:05.428954  4460 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.446053  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:05.463132  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:05.478081  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:05.481827  2280 raft_consensus.cc:2932] T 9253de3f798f4a208d566765ed2ba54d P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } }
I20240503 02:21:05.482717 18828 ts_tablet_manager.cc:902] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:05.483794  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:05.511590  2672 consensus_queue.cc:237] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:05.514381  2279 raft_consensus.cc:1261] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:05.515125 18757 consensus_queue.cc:1035] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:05.518652 18757 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:05.521670  4326 consensus_queue.cc:237] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:05.522425  2279 raft_consensus.cc:2932] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:05.524981  2412 raft_consensus.cc:1261] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 9e699d9f984b4cf2b8b468322759b78c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:05.525930 16912 consensus_queue.cc:1035] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:05.526983 18828 tablet_copy_client.cc:308] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:21:05.528139  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 51dbf0fd54b14c2fa21d3e62104c9b87 from peer 6d54feb0bbc14c109783371d2939b234 ({username='slave'} at 127.20.218.66:39913)
I20240503 02:21:05.528324  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 51dbf0fd54b14c2fa21d3e62104c9b87 from peer 6d54feb0bbc14c109783371d2939b234 at {username='slave'} at 127.20.218.66:39913: session id = 6d54feb0bbc14c109783371d2939b234-51dbf0fd54b14c2fa21d3e62104c9b87
I20240503 02:21:05.530458  2131 catalog_manager.cc:5484] T 0e61eb7d9e6041b6b99d5d5328943140 P 08055479ea104aa290d19955d3720850 reported cstate change: config changed from index 6 to 7, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } }
I20240503 02:21:05.532141  2562 tablet_copy_source_session.cc:205] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:05.540307 16912 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
W20240503 02:21:05.546062  2606 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e61eb7d9e6041b6b99d5d5328943140. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.546095  2412 raft_consensus.cc:2932] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.554277  2131 catalog_manager.cc:5484] T 592cf4d5b35e49bfa4173343d64367f4 P 6d54feb0bbc14c109783371d2939b234 reported cstate change: config changed from index 9 to 10, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New cstate: current_term: 2 leader_uuid: "9e699d9f984b4cf2b8b468322759b78c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
W20240503 02:21:05.555714  4261 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.557639  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0e61eb7d9e6041b6b99d5d5328943140 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.560997 18828 ts_tablet_manager.cc:1875] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:05.561472  2124 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 592cf4d5b35e49bfa4173343d64367f4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.563513  2540 raft_consensus.cc:2932] T a1949a5c7cc741b19da021eccb7d0dd1 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } }
I20240503 02:21:05.581490 18828 ts_tablet_manager.cc:1888] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:21:05.582029 18828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dbf0fd54b14c2fa21d3e62104c9b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:05.590023 18838 ts_tablet_manager.cc:902] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Initiating tablet copy from peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
I20240503 02:21:05.593215 18828 tablet_copy_client.cc:783] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:05.595860  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:05.609537 18828 tablet_copy_client.cc:647] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:05.612593 18828 tablet_copy_client.cc:523] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:05.618157 18828 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:21:05.628353 18838 tablet_copy_client.cc:308] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:34489
I20240503 02:21:05.630393  2301 tablet_copy_service.cc:140] P 08055479ea104aa290d19955d3720850: Received BeginTabletCopySession request for tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 08bef4c144bc4e4d8cb5836aee8da852 ({username='slave'} at 127.20.218.67:41373)
I20240503 02:21:05.630622  2301 tablet_copy_service.cc:161] P 08055479ea104aa290d19955d3720850: Beginning new tablet copy session on tablet 02e0b954008c4f30af414da6cb5d59c2 from peer 08bef4c144bc4e4d8cb5836aee8da852 at {username='slave'} at 127.20.218.67:41373: session id = 08bef4c144bc4e4d8cb5836aee8da852-02e0b954008c4f30af414da6cb5d59c2
I20240503 02:21:05.630707 18843 ts_tablet_manager.cc:902] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:05.634738  2133 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 reported cstate change: config changed from index 7 to 8, VOTER 7fd07044862a481aa387698017467c01 (127.20.218.72) evicted. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:05.635856  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:05.636631  2301 tablet_copy_source_session.cc:205] T 02e0b954008c4f30af414da6cb5d59c2 P 08055479ea104aa290d19955d3720850: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:05.643311  2672 consensus_queue.cc:237] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:05.650764 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e0b954008c4f30af414da6cb5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:21:05.666190  2607 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:05.669400 18838 tablet_copy_client.cc:783] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:05.674264  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:05.682885  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:05.687337 18838 tablet_copy_client.cc:647] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:05.690172 18838 tablet_copy_client.cc:523] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:05.705447  4326 raft_consensus.cc:1261] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Refusing update from remote peer e9fdaf326bb64edabc43b366df870f83: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:05.707463 18416 consensus_queue.cc:1035] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:05.711784  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:05.716111 18416 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.717031 18838 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:21:05.722100  4324 raft_consensus.cc:2932] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.737463  2133 catalog_manager.cc:5484] T 1eeb69e5eb3c459ca749aa37672ec057 P 9e699d9f984b4cf2b8b468322759b78c reported cstate change: config changed from index 8 to 9, VOTER 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67) added. New cstate: current_term: 1 leader_uuid: "e9fdaf326bb64edabc43b366df870f83" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:05.744307  2126 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1eeb69e5eb3c459ca749aa37672ec057 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:05.748543 18852 ts_tablet_manager.cc:902] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Initiating tablet copy from peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:05.760124  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:05.768010 18828 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:05.769410 18828 tablet_bootstrap.cc:492] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Bootstrap complete.
I20240503 02:21:05.769865 18828 ts_tablet_manager.cc:1367] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.152s	user 0.034s	sys 0.014s
I20240503 02:21:05.770942 18828 raft_consensus.cc:348] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:05.771260 18828 raft_consensus.cc:729] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:21:05.777180 18828 consensus_queue.cc:260] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:05.781827 18828 ts_tablet_manager.cc:1398] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.001s
I20240503 02:21:05.785776  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 6d54feb0bbc14c109783371d2939b234-51dbf0fd54b14c2fa21d3e62104c9b87 received from {username='slave'} at 127.20.218.66:39913
I20240503 02:21:05.786033  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 6d54feb0bbc14c109783371d2939b234-51dbf0fd54b14c2fa21d3e62104c9b87 on tablet 51dbf0fd54b14c2fa21d3e62104c9b87 with peer 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:05.805691 18859 ts_tablet_manager.cc:902] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:05.835309 18838 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:05.835814 18838 tablet_bootstrap.cc:492] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap complete.
I20240503 02:21:05.836143 18838 ts_tablet_manager.cc:1367] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.119s	user 0.024s	sys 0.003s
I20240503 02:21:05.836994 18838 raft_consensus.cc:348] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:05.837239 18838 raft_consensus.cc:729] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:21:05.842032 18843 tablet_copy_client.cc:308] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:21:05.842989  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet 8870feaf47a64a929dd2e584ece47218 from peer 08bef4c144bc4e4d8cb5836aee8da852 ({username='slave'} at 127.20.218.67:59541)
I20240503 02:21:05.843214  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet 8870feaf47a64a929dd2e584ece47218 from peer 08bef4c144bc4e4d8cb5836aee8da852 at {username='slave'} at 127.20.218.67:59541: session id = 08bef4c144bc4e4d8cb5836aee8da852-8870feaf47a64a929dd2e584ece47218
I20240503 02:21:05.847724  2432 tablet_copy_source_session.cc:205] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:05.851047 18843 ts_tablet_manager.cc:1875] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:05.858477 18838 consensus_queue.cc:260] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:05.861157  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87): 4922 bytes on disk
I20240503 02:21:05.861996 18838 ts_tablet_manager.cc:1398] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.026s	user 0.001s	sys 0.004s
I20240503 02:21:05.862784  2301 tablet_copy_service.cc:342] P 08055479ea104aa290d19955d3720850: Request end of tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-02e0b954008c4f30af414da6cb5d59c2 received from {username='slave'} at 127.20.218.67:41373
I20240503 02:21:05.862910  2301 tablet_copy_service.cc:433] P 08055479ea104aa290d19955d3720850: ending tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-02e0b954008c4f30af414da6cb5d59c2 on tablet 02e0b954008c4f30af414da6cb5d59c2 with peer 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:05.865880  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(51dbf0fd54b14c2fa21d3e62104c9b87) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 02:21:05.871381  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eeb69e5eb3c459ca749aa37672ec057: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:05.876884 18852 tablet_copy_client.cc:308] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34903
I20240503 02:21:05.877928  2432 tablet_copy_service.cc:140] P 6d54feb0bbc14c109783371d2939b234: Received BeginTabletCopySession request for tablet a7ab8b0c8db54565a6066b9a7d1569d8 from peer 08bef4c144bc4e4d8cb5836aee8da852 ({username='slave'} at 127.20.218.67:59541)
I20240503 02:21:05.878113  2432 tablet_copy_service.cc:161] P 6d54feb0bbc14c109783371d2939b234: Beginning new tablet copy session on tablet a7ab8b0c8db54565a6066b9a7d1569d8 from peer 08bef4c144bc4e4d8cb5836aee8da852 at {username='slave'} at 127.20.218.67:59541: session id = 08bef4c144bc4e4d8cb5836aee8da852-a7ab8b0c8db54565a6066b9a7d1569d8
I20240503 02:21:05.882606  2432 tablet_copy_source_session.cc:205] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:05.884011 18852 ts_tablet_manager.cc:1875] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:05.888298  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2): 4838 bytes on disk
I20240503 02:21:05.889433  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(02e0b954008c4f30af414da6cb5d59c2) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240503 02:21:05.891292 18859 tablet_copy_client.cc:308] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:21:05.891350 18843 ts_tablet_manager.cc:1888] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:21:05.891669 18843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:05.892120  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet d2351807a2ed47419f8d674fbeb9d0ec from peer e9fdaf326bb64edabc43b366df870f83 ({username='slave'} at 127.20.218.68:60819)
I20240503 02:21:05.892300  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet d2351807a2ed47419f8d674fbeb9d0ec from peer e9fdaf326bb64edabc43b366df870f83 at {username='slave'} at 127.20.218.68:60819: session id = e9fdaf326bb64edabc43b366df870f83-d2351807a2ed47419f8d674fbeb9d0ec
I20240503 02:21:05.896706  2562 tablet_copy_source_session.cc:205] T d2351807a2ed47419f8d674fbeb9d0ec P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:05.898634 18859 ts_tablet_manager.cc:1875] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:05.904254  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:05.914312  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:05.916177 18852 ts_tablet_manager.cc:1888] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:21:05.917903 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7ab8b0c8db54565a6066b9a7d1569d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:05.918296  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:05.916388 18843 tablet_copy_client.cc:783] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:05.927384 18843 tablet_copy_client.cc:647] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:05.931214 18859 ts_tablet_manager.cc:1888] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240503 02:21:05.931769 18843 tablet_copy_client.cc:523] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:05.933408 18859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2351807a2ed47419f8d674fbeb9d0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:05.942844 18852 tablet_copy_client.cc:783] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:05.954490 18852 tablet_copy_client.cc:647] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:05.958559 18852 tablet_copy_client.cc:523] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:05.961978 18859 tablet_copy_client.cc:783] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:05.962349 18843 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:21:05.963490  2541 raft_consensus.cc:1203] T 02e0b954008c4f30af414da6cb5d59c2 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:21:05.977417 18859 tablet_copy_client.cc:647] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:05.979326  2412 raft_consensus.cc:1203] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:21:05.983356 18828 ts_tablet_manager.cc:902] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:21:05.992300 18859 tablet_copy_client.cc:523] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:05.996342 18852 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:21:06.002334 18859 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Bootstrap starting.
I20240503 02:21:06.042171 18838 ts_tablet_manager.cc:902] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Initiating tablet copy from peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:21:06.065569 18828 tablet_copy_client.cc:308] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:21:06.066954  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 0e61eb7d9e6041b6b99d5d5328943140 from peer 6d54feb0bbc14c109783371d2939b234 ({username='slave'} at 127.20.218.66:53211)
I20240503 02:21:06.067130  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 0e61eb7d9e6041b6b99d5d5328943140 from peer 6d54feb0bbc14c109783371d2939b234 at {username='slave'} at 127.20.218.66:53211: session id = 6d54feb0bbc14c109783371d2939b234-0e61eb7d9e6041b6b99d5d5328943140
I20240503 02:21:06.071564  2693 tablet_copy_source_session.cc:205] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:06.073166 18828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61eb7d9e6041b6b99d5d5328943140. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:06.091240 18828 tablet_copy_client.cc:783] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:06.110316 18843 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.110875 18843 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap complete.
I20240503 02:21:06.111261 18843 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.149s	user 0.017s	sys 0.008s
I20240503 02:21:06.112227 18843 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.112531 18843 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.112826 18843 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.113700 18843 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:21:06.114593  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-8870feaf47a64a929dd2e584ece47218 received from {username='slave'} at 127.20.218.67:59541
I20240503 02:21:06.114701  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-8870feaf47a64a929dd2e584ece47218 on tablet 8870feaf47a64a929dd2e584ece47218 with peer 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:06.123625 18838 tablet_copy_client.cc:308] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.70:33893
I20240503 02:21:06.124585  4347 tablet_copy_service.cc:140] P 9e699d9f984b4cf2b8b468322759b78c: Received BeginTabletCopySession request for tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 08bef4c144bc4e4d8cb5836aee8da852 ({username='slave'} at 127.20.218.67:56343)
I20240503 02:21:06.124786  4347 tablet_copy_service.cc:161] P 9e699d9f984b4cf2b8b468322759b78c: Beginning new tablet copy session on tablet 592cf4d5b35e49bfa4173343d64367f4 from peer 08bef4c144bc4e4d8cb5836aee8da852 at {username='slave'} at 127.20.218.67:56343: session id = 08bef4c144bc4e4d8cb5836aee8da852-592cf4d5b35e49bfa4173343d64367f4
I20240503 02:21:06.129503  4347 tablet_copy_source_session.cc:205] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:06.131350 18838 ts_tablet_manager.cc:1875] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:06.136242 18852 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.136662 18852 tablet_bootstrap.cc:492] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap complete.
I20240503 02:21:06.136950 18852 ts_tablet_manager.cc:1367] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.141s	user 0.023s	sys 0.008s
I20240503 02:21:06.139217 18838 ts_tablet_manager.cc:1888] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.7
I20240503 02:21:06.139693 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 592cf4d5b35e49bfa4173343d64367f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:06.142554  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218): 4923 bytes on disk
I20240503 02:21:06.143194  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.146021 18828 tablet_copy_client.cc:647] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:06.154182 18838 tablet_copy_client.cc:783] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:06.154466  2541 raft_consensus.cc:1203] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:21:06.159116 18852 raft_consensus.cc:348] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.161931 18852 raft_consensus.cc:729] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.163092 18852 consensus_queue.cc:260] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.164237 18828 tablet_copy_client.cc:523] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:06.164880 18852 ts_tablet_manager.cc:1398] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.000s
I20240503 02:21:06.167500  2432 tablet_copy_service.cc:342] P 6d54feb0bbc14c109783371d2939b234: Request end of tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-a7ab8b0c8db54565a6066b9a7d1569d8 received from {username='slave'} at 127.20.218.67:59541
I20240503 02:21:06.167635  2432 tablet_copy_service.cc:433] P 6d54feb0bbc14c109783371d2939b234: ending tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-a7ab8b0c8db54565a6066b9a7d1569d8 on tablet a7ab8b0c8db54565a6066b9a7d1569d8 with peer 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:06.171685 18838 tablet_copy_client.cc:647] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:06.175459 18838 tablet_copy_client.cc:523] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:06.183378 18828 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Bootstrap starting.
I20240503 02:21:06.183827 18838 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:21:06.196566 18859 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.197211 18859 tablet_bootstrap.cc:492] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Bootstrap complete.
I20240503 02:21:06.197609 18859 ts_tablet_manager.cc:1367] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Time spent bootstrapping tablet: real 0.195s	user 0.046s	sys 0.014s
I20240503 02:21:06.198623 18859 raft_consensus.cc:348] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:21:06.198870 18859 raft_consensus.cc:729] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9fdaf326bb64edabc43b366df870f83, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.199139 18859 consensus_queue.cc:260] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } }
I20240503 02:21:06.199471 18852 ts_tablet_manager.cc:902] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Initiating tablet copy from peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103)
I20240503 02:21:06.199853 18859 ts_tablet_manager.cc:1398] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:06.206041  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session e9fdaf326bb64edabc43b366df870f83-d2351807a2ed47419f8d674fbeb9d0ec received from {username='slave'} at 127.20.218.68:60819
I20240503 02:21:06.206238  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session e9fdaf326bb64edabc43b366df870f83-d2351807a2ed47419f8d674fbeb9d0ec on tablet d2351807a2ed47419f8d674fbeb9d0ec with peer e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:06.227820 18838 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.230000 18838 tablet_bootstrap.cc:492] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap complete.
I20240503 02:21:06.230338 18838 ts_tablet_manager.cc:1367] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.047s	user 0.030s	sys 0.003s
I20240503 02:21:06.232029 18838 raft_consensus.cc:348] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.232295 18838 raft_consensus.cc:729] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.232554 18838 consensus_queue.cc:260] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.237124 18838 ts_tablet_manager.cc:1398] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240503 02:21:06.239096  4347 tablet_copy_service.cc:342] P 9e699d9f984b4cf2b8b468322759b78c: Request end of tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-592cf4d5b35e49bfa4173343d64367f4 received from {username='slave'} at 127.20.218.67:56343
I20240503 02:21:06.239315  4347 tablet_copy_service.cc:433] P 9e699d9f984b4cf2b8b468322759b78c: ending tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-592cf4d5b35e49bfa4173343d64367f4 on tablet 592cf4d5b35e49bfa4173343d64367f4 with peer 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:06.246778 18828 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.247251 18828 tablet_bootstrap.cc:492] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Bootstrap complete.
I20240503 02:21:06.247573 18828 ts_tablet_manager.cc:1367] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Time spent bootstrapping tablet: real 0.064s	user 0.020s	sys 0.007s
I20240503 02:21:06.248421  2672 raft_consensus.cc:1203] T d2351807a2ed47419f8d674fbeb9d0ec P e9fdaf326bb64edabc43b366df870f83 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240503 02:21:06.248498 18828 raft_consensus.cc:348] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:06.248718 18828 raft_consensus.cc:729] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.249689 18852 tablet_copy_client.cc:308] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.68:38103
I20240503 02:21:06.249788 18828 consensus_queue.cc:260] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:06.251056 18828 ts_tablet_manager.cc:1398] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:21:06.251291  2693 tablet_copy_service.cc:140] P e9fdaf326bb64edabc43b366df870f83: Received BeginTabletCopySession request for tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 08bef4c144bc4e4d8cb5836aee8da852 ({username='slave'} at 127.20.218.67:40091)
I20240503 02:21:06.251441  2693 tablet_copy_service.cc:161] P e9fdaf326bb64edabc43b366df870f83: Beginning new tablet copy session on tablet 1eeb69e5eb3c459ca749aa37672ec057 from peer 08bef4c144bc4e4d8cb5836aee8da852 at {username='slave'} at 127.20.218.67:40091: session id = 08bef4c144bc4e4d8cb5836aee8da852-1eeb69e5eb3c459ca749aa37672ec057
I20240503 02:21:06.255831  2692 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 6d54feb0bbc14c109783371d2939b234-0e61eb7d9e6041b6b99d5d5328943140 received from {username='slave'} at 127.20.218.66:53211
I20240503 02:21:06.256026  2692 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 6d54feb0bbc14c109783371d2939b234-0e61eb7d9e6041b6b99d5d5328943140 on tablet 0e61eb7d9e6041b6b99d5d5328943140 with peer 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:06.257467  2693 tablet_copy_source_session.cc:205] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:06.259114 18852 ts_tablet_manager.cc:1875] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:06.271939 18852 ts_tablet_manager.cc:1888] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240503 02:21:06.272306 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eeb69e5eb3c459ca749aa37672ec057. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:06.277127 18852 tablet_copy_client.cc:783] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:06.284953 18852 tablet_copy_client.cc:647] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:06.287606 18852 tablet_copy_client.cc:523] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:06.294788 18852 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap starting.
I20240503 02:21:06.330632  2541 raft_consensus.cc:1203] T a7ab8b0c8db54565a6066b9a7d1569d8 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:21:06.356163 18852 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:06.356637 18852 tablet_bootstrap.cc:492] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Bootstrap complete.
I20240503 02:21:06.356974 18852 ts_tablet_manager.cc:1367] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent bootstrapping tablet: real 0.062s	user 0.051s	sys 0.001s
I20240503 02:21:06.357746 18852 raft_consensus.cc:348] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.357973 18852 raft_consensus.cc:729] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Initialized, Role: FOLLOWER
I20240503 02:21:06.358201 18852 consensus_queue.cc:260] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:06.358727 18852 ts_tablet_manager.cc:1398] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240503 02:21:06.359562  2693 tablet_copy_service.cc:342] P e9fdaf326bb64edabc43b366df870f83: Request end of tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-1eeb69e5eb3c459ca749aa37672ec057 received from {username='slave'} at 127.20.218.67:40091
I20240503 02:21:06.359696  2693 tablet_copy_service.cc:433] P e9fdaf326bb64edabc43b366df870f83: ending tablet copy session 08bef4c144bc4e4d8cb5836aee8da852-1eeb69e5eb3c459ca749aa37672ec057 on tablet 1eeb69e5eb3c459ca749aa37672ec057 with peer 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:06.370358  2458 maintenance_manager.cc:419] P 6d54feb0bbc14c109783371d2939b234: Scheduling UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140): 4583 bytes on disk
I20240503 02:21:06.371399  2349 maintenance_manager.cc:643] P 6d54feb0bbc14c109783371d2939b234: UndoDeltaBlockGCOp(0e61eb7d9e6041b6b99d5d5328943140) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.377475  2719 maintenance_manager.cc:419] P e9fdaf326bb64edabc43b366df870f83: Scheduling UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec): 4583 bytes on disk
I20240503 02:21:06.378165  2610 maintenance_manager.cc:643] P e9fdaf326bb64edabc43b366df870f83: UndoDeltaBlockGCOp(d2351807a2ed47419f8d674fbeb9d0ec) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.395948  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8): 4923 bytes on disk
I20240503 02:21:06.396684  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(a7ab8b0c8db54565a6066b9a7d1569d8) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.398120  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057): 4922 bytes on disk
I20240503 02:21:06.398769  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(1eeb69e5eb3c459ca749aa37672ec057) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.400247  2588 maintenance_manager.cc:419] P 08bef4c144bc4e4d8cb5836aee8da852: Scheduling UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4): 4583 bytes on disk
I20240503 02:21:06.400846  2479 maintenance_manager.cc:643] P 08bef4c144bc4e4d8cb5836aee8da852: UndoDeltaBlockGCOp(592cf4d5b35e49bfa4173343d64367f4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 02:21:06.462603  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:06.536672  2412 raft_consensus.cc:1203] T 0e61eb7d9e6041b6b99d5d5328943140 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240503 02:21:06.567955  2607 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 63936.9us
I20240503 02:21:06.584360  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:06.620185  2541 raft_consensus.cc:1203] T 1eeb69e5eb3c459ca749aa37672ec057 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240503 02:21:06.646697  2541 raft_consensus.cc:1203] T 592cf4d5b35e49bfa4173343d64367f4 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240503 02:21:06.655117  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:06.663754  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:06.675812  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:06.704082  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:06.711907  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240503 02:21:06.872196  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eeb69e5eb3c459ca749aa37672ec057: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:06.954154  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:06.974092  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:06.974452  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:06.992720  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:07.055692  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:07.074753  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:07.143810  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:07.287068  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:07.311162  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:07.321375  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:07.321585  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:07.332912  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:07.365005  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:07.430045  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:07.716295  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:07.733490  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:07.750782  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:07.771113  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:07.802538  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:07.815068  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:07.819267  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:07.909574  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1eeb69e5eb3c459ca749aa37672ec057: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:07.979027  4643 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:15:15.043944  4652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:15.044776  4652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:15.058809  4652 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:18.560497  4652 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:15:20.973214  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.973809  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.973999  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:20.974138  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:20.974296  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:20.974465  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.974612  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:20.974774  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:20.974989  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:20.975165  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.975281  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:20.975440  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.975584  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:20.975719  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:20.975821  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:21.347709  4652 rebalancer_tool.cc:1074] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:21.928583  4652 rebalancer_tool.cc:1074] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:22.742697  4652 rebalancer_tool.cc:1074] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:23.471773  4652 rebalancer_tool.cc:1074] tablet 02e0b954008c4f30af414da6cb5d59c2: '6d54feb0bbc14c109783371d2939b234' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:24.100224  4652 rebalancer_tool.cc:1074] tablet 1ec063c94fe941ebb1c866b5b8935ece: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:24.589201  4652 rebalancer_tool.cc:1084] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240503 02:15:24.589447  4652 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:26.260040  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:26.260448  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 7fd07044862a481aa387698017467c01 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:26.260690  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:26.260824  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:26.260926  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:26.261054  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:26.261157  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:26.261318  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:26.261732  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:26.261955  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:26.262112  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:26.262277  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:26.449699  4652 rebalancer_tool.cc:1074] tablet 1eeb69e5eb3c459ca749aa37672ec057: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:26.511482  4652 rebalancer_tool.cc:1084] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move ignored: Illegal state: tablet 9f7c9c29384949fb95cb0c3b765f57b0: replica 408af0ab5520465e942cc714bfc6c73b is already present
I20240503 02:15:26.511634  4652 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:28.749451  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.749832  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:28.750080  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:28.750209  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:28.750330  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.750455  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:28.750603  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.750850  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:28.751093  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:28.751242  4652 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:28.908440  4652 rebalancer_tool.cc:1074] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:29.609062  4652 rebalancer_tool.cc:1074] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:30.647814  4652 rebalancer_tool.cc:1074] tablet 7a62b3ef44e94e92b0feed9c228f5702: '08bef4c144bc4e4d8cb5836aee8da852' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:30.695964  4652 rebalancer_tool.cc:1074] tablet 734027e195414b15811ac9d32b132db8: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:31.200837  4652 rebalancer_tool.cc:1074] tablet 913d0aa014ad4bcaac905b5fdaa296d2: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:31.368381  4652 rebalancer_tool.cc:1074] tablet 771a4190142b4fd5a9479417122c0881: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:31.535223  4652 rebalancer_tool.cc:1074] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:39.846690  4652 rebalancer_tool.cc:1136] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:39.936151  4652 rebalancer_tool.cc:1136] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:15:44.477865  4652 rebalancer_tool.cc:1136] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
W20240503 02:15:45.762858  9057 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.Quiesce -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 48519.7us
I20240503 02:15:49.088569  4652 rebalancer_tool.cc:1136] tablet 1ec063c94fe941ebb1c866b5b8935ece: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:49.430831  4652 rebalancer_tool.cc:1136] tablet 734027e195414b15811ac9d32b132db8: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:49.622329  4652 rebalancer_tool.cc:1136] tablet 1eeb69e5eb3c459ca749aa37672ec057: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:49.708221  4652 rebalancer_tool.cc:1074] tablet eadcb49d12664e6fb5a5ad8dbe47f342: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:49.904752  4652 rebalancer_tool.cc:1074] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:50.095299  4652 rebalancer_tool.cc:1074] tablet d2351807a2ed47419f8d674fbeb9d0ec: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:54.053989  4652 rebalancer_tool.cc:1136] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:15:54.273526  4652 rebalancer_tool.cc:1136] tablet eadcb49d12664e6fb5a5ad8dbe47f342: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:54.384773  4652 rebalancer_tool.cc:1136] tablet 771a4190142b4fd5a9479417122c0881: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:54.629379  4652 rebalancer_tool.cc:1136] tablet 913d0aa014ad4bcaac905b5fdaa296d2: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:54.858916  4652 rebalancer_tool.cc:1136] tablet 7a62b3ef44e94e92b0feed9c228f5702: '08bef4c144bc4e4d8cb5836aee8da852' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:55.066918  4652 rebalancer_tool.cc:1136] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:55.264379  4652 rebalancer_tool.cc:1136] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:55.334965  4652 rebalancer_tool.cc:1136] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:55.384439  4652 rebalancer_tool.cc:1136] tablet 02e0b954008c4f30af414da6cb5d59c2: '6d54feb0bbc14c109783371d2939b234' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:55.534574  4652 rebalancer_tool.cc:1074] tablet 0e61eb7d9e6041b6b99d5d5328943140: '7fd07044862a481aa387698017467c01' -> '' move scheduled
I20240503 02:15:55.643821  4652 rebalancer_tool.cc:1074] tablet 0137e646f1bd452b950d870107afa060: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:55.764068  4652 rebalancer_tool.cc:1074] tablet 62a3dcc2de594c8982477a0229f71191: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:55.908107  4652 rebalancer_tool.cc:1074] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:56.046730  4652 rebalancer_tool.cc:1136] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
W20240503 02:16:05.969406 12406 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:16:05.859271 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.218.72:37033 (local address 127.0.0.1:50072)
0503 02:16:05.859507 (+   236us) negotiation.cc:107] Waiting for socket to connect
0503 02:16:05.859529 (+    22us) client_negotiation.cc:174] Beginning negotiation
0503 02:16:05.859652 (+   123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:16:05.967896 (+108244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130}
I20240503 02:16:06.654573  4652 rebalancer_tool.cc:1136] tablet d2351807a2ed47419f8d674fbeb9d0ec: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:06.815529  4652 rebalancer_tool.cc:1074] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:07.054381  4652 rebalancer_tool.cc:1074] tablet 8870feaf47a64a929dd2e584ece47218: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:16:07.179311  4652 rebalancer_tool.cc:1136] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:07.233712  4652 rebalancer_tool.cc:1136] tablet 62a3dcc2de594c8982477a0229f71191: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:07.351667  4652 rebalancer_tool.cc:1136] tablet 8870feaf47a64a929dd2e584ece47218: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:07.465147  4652 rebalancer_tool.cc:1136] tablet 0137e646f1bd452b950d870107afa060: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
W20240503 02:16:08.543058 12829 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.20.218.126:35007, user_credentials={real_user=slave}} blocked reactor thread for 44160.1us
W20240503 02:19:01.135504  1918 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.GetFlags -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 60683.2us
W20240503 02:19:37.661499  7648 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.68:38103, user_credentials={real_user=slave}} blocked reactor thread for 51789.9us
I20240503 02:21:05.694593  4652 rebalancer_tool.cc:1136] tablet 0e61eb7d9e6041b6b99d5d5328943140: '7fd07044862a481aa387698017467c01' -> '' move completed: OK
Illegal state: tablet server 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): unacceptable health status UNAVAILABLE

I20240503 02:21:07.989503  2410 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218"
dest_uuid: "6d54feb0bbc14c109783371d2939b234"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56208
I20240503 02:21:07.989758  2410 raft_consensus.cc:595] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:08.138113  2521 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:08.170526  2390 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:08.176676  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:08.178349  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:08.189774  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:08.213845  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:08.240599  4647 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:15:14.967454  4648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:14.968062  4648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:14.982905  4648 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 02:15:15.674521  4689 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.20.218.126:35007, user_credentials={real_user=slave}} blocked reactor thread for 44220.6us
I20240503 02:15:17.957420  4648 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:15:19.839828  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:19.841944  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:19.842504  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:19.842832  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:19.843571  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:19.845372  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:19.845726  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:19.846062  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:19.846424  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:19.846719  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:19.847052  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:19.847332  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:19.849061  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:19.849438  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:19.850214  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:20.820398  4648 rebalancer_tool.cc:1074] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:21.124125  4648 rebalancer_tool.cc:1074] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:21.649567  4648 rebalancer_tool.cc:1074] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:22.338392  4648 rebalancer_tool.cc:1074] tablet 62a3dcc2de594c8982477a0229f71191: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:22.843149  4648 rebalancer_tool.cc:1074] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:23.562268  4648 rebalancer_tool.cc:1074] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:24.180238  4648 rebalancer_tool.cc:1074] tablet 734027e195414b15811ac9d32b132db8: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:25.138620  4648 rebalancer_tool.cc:1074] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:25.619274  4648 rebalancer_tool.cc:1074] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:26.170095  4648 rebalancer_tool.cc:1074] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:26.698987  4648 rebalancer_tool.cc:1074] tablet 771a4190142b4fd5a9479417122c0881: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:27.453538  4648 rebalancer_tool.cc:1074] tablet 592cf4d5b35e49bfa4173343d64367f4: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:28.083205  4648 rebalancer_tool.cc:1074] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:28.642184  4648 rebalancer_tool.cc:1074] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:29.157445  4648 rebalancer_tool.cc:1074] tablet 9b822c691d824b2ba25e967535c9734e: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:29.820505  4648 rebalancer_tool.cc:1074] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:30.256986  4648 rebalancer_tool.cc:1084] tablet 913d0aa014ad4bcaac905b5fdaa296d2: '08055479ea104aa290d19955d3720850' -> '1d36807868fe4a82bca42c25831ce8a2' move ignored: Illegal state: tablet 913d0aa014ad4bcaac905b5fdaa296d2: replica 1d36807868fe4a82bca42c25831ce8a2 is already present
I20240503 02:15:30.257275  4648 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:32.029633  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.034379  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 408af0ab5520465e942cc714bfc6c73b to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:32.034926  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:32.035239  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.036000  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:32.037864  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.038493  4648 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:32.260052  4648 rebalancer_tool.cc:1074] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:32.338850  4648 rebalancer_tool.cc:1074] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:32.440943  4648 rebalancer_tool.cc:1074] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:32.548959  4648 rebalancer_tool.cc:1074] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move scheduled
I20240503 02:15:32.632141  4648 rebalancer_tool.cc:1074] tablet ae239d73f0bc4d9aac025e40489e81a6: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:35.123802  4648 rebalancer_tool.cc:1136] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move completed: OK
I20240503 02:15:37.746882  4648 rebalancer_tool.cc:1136] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:43.098107  4648 rebalancer_tool.cc:1136] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:45.474265  4648 rebalancer_tool.cc:1123] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move is abandoned: Not found: Server with UUID 08055479ea104aa290d19955d3720850 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "7fd07044862a481aa387698017467c01" member_type: VOTER last_known_addr { host: "127.20.218.72" port: 37033 } }
I20240503 02:15:45.524171  4648 rebalancer_tool.cc:1136] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:48.333307  4648 rebalancer_tool.cc:1136] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:48.392410  4648 rebalancer_tool.cc:1136] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:48.536108  4648 rebalancer_tool.cc:1136] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move completed: OK
W20240503 02:15:49.322539  9757 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s	user 0.000s	sys 0.000s
I20240503 02:15:50.786175  4648 rebalancer_tool.cc:1136] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:15:51.015201  4648 rebalancer_tool.cc:1136] tablet 771a4190142b4fd5a9479417122c0881: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:55.843345  4648 rebalancer_tool.cc:1136] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:16:00.589797  4648 rebalancer_tool.cc:1136] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:16:00.691972  4648 rebalancer_tool.cc:1074] tablet 8870feaf47a64a929dd2e584ece47218: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:00.768182  4648 rebalancer_tool.cc:1074] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:16:00.904362  4648 rebalancer_tool.cc:1074] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:16:00.967911  4648 rebalancer_tool.cc:1136] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:16:01.132263  4648 rebalancer_tool.cc:1136] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:16:01.523869  4648 rebalancer_tool.cc:1136] tablet 8870feaf47a64a929dd2e584ece47218: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:01.630964  4648 rebalancer_tool.cc:1136] tablet ae239d73f0bc4d9aac025e40489e81a6: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:01.932647  4648 rebalancer_tool.cc:1136] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:16:02.044624  4648 rebalancer_tool.cc:1136] tablet 62a3dcc2de594c8982477a0229f71191: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:16:02.127594  4648 rebalancer_tool.cc:1136] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:16:02.273707  4648 rebalancer_tool.cc:1136] tablet 734027e195414b15811ac9d32b132db8: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:16:02.399144  4648 rebalancer_tool.cc:1136] tablet 9f7c9c29384949fb95cb0c3b765f57b0: '08bef4c144bc4e4d8cb5836aee8da852' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:16:02.499500  4648 rebalancer_tool.cc:1136] tablet 75ba33132e784f408f3e26882aea3d15: '08055479ea104aa290d19955d3720850' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:16:04.584482  4648 rebalancer_tool.cc:1136] tablet 9b822c691d824b2ba25e967535c9734e: 'e9fdaf326bb64edabc43b366df870f83' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:16:04.760824  4648 rebalancer_tool.cc:1074] tablet 586ebf467f0f427fa62dee6223a39848: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:16:09.860618  4648 rebalancer_tool.cc:1136] tablet 586ebf467f0f427fa62dee6223a39848: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
W20240503 02:16:32.903618 16060 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.126:35007, user_credentials={real_user=slave}} blocked reactor thread for 45304.6us
W20240503 02:19:51.961208  9407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.69:39741, user_credentials={real_user=slave}} blocked reactor thread for 68405.6us
W20240503 02:20:23.661417 13472 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s	user 0.001s	sys 0.000s
I20240503 02:21:06.646970  4648 rebalancer_tool.cc:1136] tablet 592cf4d5b35e49bfa4173343d64367f4: '6d54feb0bbc14c109783371d2939b234' -> '7fd07044862a481aa387698017467c01' move completed: Incomplete: tablet 592cf4d5b35e49bfa4173343d64367f4, TS 6d54feb0bbc14c109783371d2939b234 -> TS 7fd07044862a481aa387698017467c01 move failed, target replica disappeared
Illegal state: tablet server 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): unacceptable health status UNAVAILABLE

I20240503 02:21:08.250483  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:08.297565 18777 raft_consensus.cc:979] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: : Instructing follower 08bef4c144bc4e4d8cb5836aee8da852 to start an election
I20240503 02:21:08.297852 18777 raft_consensus.cc:1067] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Signalling peer 08bef4c144bc4e4d8cb5836aee8da852 to start an election
I20240503 02:21:08.298696  2541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218"
dest_uuid: "08bef4c144bc4e4d8cb5836aee8da852"
 from {username='slave'} at 127.20.218.66:39913
I20240503 02:21:08.298940  2541 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:21:08.299058  2541 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:08.302201  2541 raft_consensus.cc:504] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:08.303351  2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "6d54feb0bbc14c109783371d2939b234"
I20240503 02:21:08.303575  2410 raft_consensus.cc:3032] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:21:08.303656  2410 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6d54feb0bbc14c109783371d2939b234, State: Running, Role: LEADER
I20240503 02:21:08.303941  2410 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:08.304273  2410 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:08.307444  2410 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 2.
I20240503 02:21:08.309763  2475 leader_election.cc:302] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234; no voters: 
I20240503 02:21:08.310135 18562 raft_consensus.cc:2781] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:21:08.310295 18562 raft_consensus.cc:686] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Becoming Leader. State: Replica: 08bef4c144bc4e4d8cb5836aee8da852, State: Running, Role: LEADER
I20240503 02:21:08.310576 18562 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:08.310725  2541 leader_election.cc:288] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [CANDIDATE]: Term 2 election: Requested vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:08.310842  4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "08bef4c144bc4e4d8cb5836aee8da852" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "408af0ab5520465e942cc714bfc6c73b"
I20240503 02:21:08.311058  4461 raft_consensus.cc:3037] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:08.314244  4461 raft_consensus.cc:2454] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bef4c144bc4e4d8cb5836aee8da852 in term 2.
I20240503 02:21:08.314483  2132 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: term changed from 1 to 2, leader changed from 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) to 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:08.652921  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:08.655432  2521 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:08.659636  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:08.671577  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:08.684978  2390 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:08.704574  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:08.711396  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:08.735208  2410 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:08.735854 18562 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:08.800084  4461 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:08.803077 18562 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:09.203375  2521 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:09.210798  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:09.231629  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:09.241405  2390 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:09.253726  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:09.282939  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 02:21:09.289057  2126 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.218.72:37033, user_credentials={real_user=slave}} blocked reactor thread for 52841.1us
W20240503 02:21:09.290616  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51dbf0fd54b14c2fa21d3e62104c9b87: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
W20240503 02:21:09.299551  4132 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 78256.5us
I20240503 02:21:09.324877  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:09.431289  2541 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:09.432922  4461 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:21:09.433432 18562 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:21:09.435531 19196 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:09.436208  4461 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:09.442327  2135 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 9 to 11, VOTER 6d54feb0bbc14c109783371d2939b234 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } }
I20240503 02:21:09.452711  2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 8870feaf47a64a929dd2e584ece47218 with delete_type TABLET_DATA_TOMBSTONED (TS 6d54feb0bbc14c109783371d2939b234 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56126
I20240503 02:21:09.453397 19217 tablet_replica.cc:330] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: stopping tablet replica
I20240503 02:21:09.453619 19217 raft_consensus.cc:2229] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:09.453811 19217 raft_consensus.cc:2258] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:09.454080  2541 consensus_queue.cc:237] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:09.455013 19217 ts_tablet_manager.cc:1875] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:21:09.455968  2475 consensus_peers.cc:485] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893): Couldn't send request to peer 9e699d9f984b4cf2b8b468322759b78c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8870feaf47a64a929dd2e584ece47218. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:09.455732  4461 raft_consensus.cc:1261] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Refusing update from remote peer 08bef4c144bc4e4d8cb5836aee8da852: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240503 02:21:09.456598 19196 consensus_queue.cc:1035] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 02:21:09.459015 18562 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:09.462594  2125 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8870feaf47a64a929dd2e584ece47218 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240503 02:21:09.463922  4460 raft_consensus.cc:2932] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:09.467258 19217 ts_tablet_manager.cc:1888] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:21:09.467411 19217 log.cc:1196] T 8870feaf47a64a929dd2e584ece47218 P 6d54feb0bbc14c109783371d2939b234: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/8870feaf47a64a929dd2e584ece47218
I20240503 02:21:09.467371  2138 catalog_manager.cc:5484] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852 reported cstate change: config changed from index 11 to 12, VOTER 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70) added. New cstate: current_term: 2 leader_uuid: "08bef4c144bc4e4d8cb5836aee8da852" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } }
I20240503 02:21:09.468201  2124 catalog_manager.cc:4830] TS 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): tablet 8870feaf47a64a929dd2e584ece47218 (table rebalance_test_table_0 [id=92ee3a87e7c743bd87f764904e01b8d8]) successfully deleted
I20240503 02:21:09.808282  2521 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:21:09.827473  4441 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:09.863279  2261 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:21:09.875595  2390 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:09.882248  4176 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240503 02:21:09.886163  2348 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 62409.2us
I20240503 02:21:09.914717  4307 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:21:09.924352  2651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:09.950975 19271 ts_tablet_manager.cc:902] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Initiating tablet copy from peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
W20240503 02:21:09.976408  2126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111) [suppressed 115 similar messages]
I20240503 02:21:10.030025 19271 tablet_copy_client.cc:308] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:38127
I20240503 02:21:10.030854  2562 tablet_copy_service.cc:140] P 08bef4c144bc4e4d8cb5836aee8da852: Received BeginTabletCopySession request for tablet 8870feaf47a64a929dd2e584ece47218 from peer 9e699d9f984b4cf2b8b468322759b78c ({username='slave'} at 127.20.218.70:51849)
I20240503 02:21:10.031050  2562 tablet_copy_service.cc:161] P 08bef4c144bc4e4d8cb5836aee8da852: Beginning new tablet copy session on tablet 8870feaf47a64a929dd2e584ece47218 from peer 9e699d9f984b4cf2b8b468322759b78c at {username='slave'} at 127.20.218.70:51849: session id = 9e699d9f984b4cf2b8b468322759b78c-8870feaf47a64a929dd2e584ece47218
I20240503 02:21:10.034683  2562 tablet_copy_source_session.cc:205] T 8870feaf47a64a929dd2e584ece47218 P 08bef4c144bc4e4d8cb5836aee8da852: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 02:21:10.036202 19271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8870feaf47a64a929dd2e584ece47218. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:10.042248 19271 tablet_copy_client.cc:783] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 5 data blocks...
I20240503 02:21:10.049046 19271 tablet_copy_client.cc:647] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:10.051803 19271 tablet_copy_client.cc:523] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:10.055524 19271 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap starting.
I20240503 02:21:10.076872 19271 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:10.077373 19271 tablet_bootstrap.cc:492] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Bootstrap complete.
I20240503 02:21:10.077701 19271 ts_tablet_manager.cc:1367] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Time spent bootstrapping tablet: real 0.022s	user 0.020s	sys 0.003s
I20240503 02:21:10.078534 19271 raft_consensus.cc:348] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:10.078769 19271 raft_consensus.cc:729] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e699d9f984b4cf2b8b468322759b78c, State: Initialized, Role: FOLLOWER
I20240503 02:21:10.079020 19271 consensus_queue.cc:260] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:10.079576 19271 ts_tablet_manager.cc:1398] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:10.080605  2562 tablet_copy_service.cc:342] P 08bef4c144bc4e4d8cb5836aee8da852: Request end of tablet copy session 9e699d9f984b4cf2b8b468322759b78c-8870feaf47a64a929dd2e584ece47218 received from {username='slave'} at 127.20.218.70:51849
I20240503 02:21:10.080829  2562 tablet_copy_service.cc:433] P 08bef4c144bc4e4d8cb5836aee8da852: ending tablet copy session 9e699d9f984b4cf2b8b468322759b78c-8870feaf47a64a929dd2e584ece47218 on tablet 8870feaf47a64a929dd2e584ece47218 with peer 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:21:10.215730  4373 maintenance_manager.cc:419] P 9e699d9f984b4cf2b8b468322759b78c: Scheduling UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218): 4923 bytes on disk
I20240503 02:21:10.216343  4264 maintenance_manager.cc:643] P 9e699d9f984b4cf2b8b468322759b78c: UndoDeltaBlockGCOp(8870feaf47a64a929dd2e584ece47218) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 02:21:10.472802  4326 raft_consensus.cc:1203] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
I20240503 02:21:10.473747  4646 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240503 02:15:15.427690  4650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:15:15.437539  4650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:15:15.451119  4650 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:15:19.240216  4650 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 02:15:20.462278  5317 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.20.218.126:35007, user_credentials={real_user=slave}} blocked reactor thread for 104363us
W20240503 02:15:21.097622  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.097846  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.098004  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.098196  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.098307  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.098429  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.098697  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.098920  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:21.099037  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.099210  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.099352  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.099576  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.099748  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:21.099881  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:21.100008  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:21.100121  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:21.806047  4650 rebalancer_tool.cc:1074] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:22.623034  4650 rebalancer_tool.cc:1074] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:23.231284  4650 rebalancer_tool.cc:1074] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:23.343482  4650 rebalancer_tool.cc:1084] tablet d2351807a2ed47419f8d674fbeb9d0ec: 'e9fdaf326bb64edabc43b366df870f83' -> '7fd07044862a481aa387698017467c01' move ignored: Illegal state: tablet d2351807a2ed47419f8d674fbeb9d0ec: replica 7fd07044862a481aa387698017467c01 is already present
I20240503 02:15:23.343676  4650 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:25.095562  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:25.095880  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92e4b9e7a87649ed8114a3f011cc6b56: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:25.096036  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:25.096174  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:25.096267  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:25.096369  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:25.096510  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:25.096637  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:25.096746  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:25.096833  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 884eb53e27af4de69b5921029b1ca76e: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:25.096947  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:25.097047  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:25.097159  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:25.097254  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:25.097456  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:25.097563  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:25.097693  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:25.097782  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:15:25.315202  4650 rebalancer_tool.cc:1074] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:25.451128  4650 rebalancer_tool.cc:1074] tablet 62a3dcc2de594c8982477a0229f71191: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:25.849305  4650 rebalancer_tool.cc:1074] tablet 9b822c691d824b2ba25e967535c9734e: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move scheduled
I20240503 02:15:26.285281  4650 rebalancer_tool.cc:1074] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '408af0ab5520465e942cc714bfc6c73b' move scheduled
I20240503 02:15:26.606910  4650 rebalancer_tool.cc:1084] tablet 771a4190142b4fd5a9479417122c0881: '08055479ea104aa290d19955d3720850' -> '1d36807868fe4a82bca42c25831ce8a2' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:15:26.607080  4650 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:28.566681  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:28.576593  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.578102  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.578516  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:28.580976  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4d368e5356e14979916b32fbd558c44a: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:28.581506  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 92ee3a87e7c743bd87f764904e01b8d8: could not find any suitable replica to move from server 6d54feb0bbc14c109783371d2939b234 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:28.582203  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc0f9aa11af74e438eac8b71019ea20b: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 9e699d9f984b4cf2b8b468322759b78c
W20240503 02:15:28.583878  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:28.584043  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server 08bef4c144bc4e4d8cb5836aee8da852 to server 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:15:28.743285  4650 rebalancer_tool.cc:1074] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move scheduled
I20240503 02:15:29.403723  4650 rebalancer_tool.cc:1074] tablet 913d0aa014ad4bcaac905b5fdaa296d2: '08055479ea104aa290d19955d3720850' -> '1d36807868fe4a82bca42c25831ce8a2' move scheduled
I20240503 02:15:29.623265  4650 rebalancer_tool.cc:1074] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:30.892611  4650 rebalancer_tool.cc:1074] tablet a7ab8b0c8db54565a6066b9a7d1569d8: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move scheduled
I20240503 02:15:31.010711  4650 rebalancer_tool.cc:1084] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '9e699d9f984b4cf2b8b468322759b78c' move ignored: Illegal state: tablet 43e248f0cb764d229802851c4e73cc96: replica 9e699d9f984b4cf2b8b468322759b78c is already present
I20240503 02:15:31.010900  4650 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 02:15:32.789023  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 263a2ae2aba54460aee371a910b0c410: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 1d36807868fe4a82bca42c25831ce8a2
W20240503 02:15:32.789302  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e6f074dfde9442ee808f069f0aa37634: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.789458  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7ef4fa73f50841faa48a3fd6ec9d64a6: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 408af0ab5520465e942cc714bfc6c73b
W20240503 02:15:32.803061  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 448499f38b724268990226d70f70f241: could not find any suitable replica to move from server e9fdaf326bb64edabc43b366df870f83 to server 7fd07044862a481aa387698017467c01
W20240503 02:15:32.803326  4650 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2d127e4ea62a4b498d1efd3a86c450c6: could not find any suitable replica to move from server 08055479ea104aa290d19955d3720850 to server 408af0ab5520465e942cc714bfc6c73b
I20240503 02:15:32.937618  4650 rebalancer_tool.cc:1074] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:33.089537  4650 rebalancer_tool.cc:1074] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move scheduled
I20240503 02:15:35.464040  4650 rebalancer_tool.cc:1136] tablet 1ec063c94fe941ebb1c866b5b8935ece: '7fd07044862a481aa387698017467c01' -> '' move completed: OK
I20240503 02:15:42.845232  4650 rebalancer_tool.cc:1136] tablet 02e0b954008c4f30af414da6cb5d59c2: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:43.136854  4650 rebalancer_tool.cc:1136] tablet a1949a5c7cc741b19da021eccb7d0dd1: '6d54feb0bbc14c109783371d2939b234' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:45.494745  4650 rebalancer_tool.cc:1136] tablet 913d0aa014ad4bcaac905b5fdaa296d2: '08055479ea104aa290d19955d3720850' -> '1d36807868fe4a82bca42c25831ce8a2' move completed: OK
I20240503 02:15:45.664461  4650 rebalancer_tool.cc:1136] tablet 2de1d29dbea945eba58f8eadb019d766: '08055479ea104aa290d19955d3720850' -> '7fd07044862a481aa387698017467c01' move completed: OK
W20240503 02:15:49.667222  9599 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.ServerClock -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 68747us
W20240503 02:15:49.732760  9599 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.71:38285, user_credentials={real_user=slave}} blocked reactor thread for 48226.6us
I20240503 02:15:55.606771  4650 rebalancer_tool.cc:1074] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:55.800312  4650 rebalancer_tool.cc:1074] tablet f0a088d230d0462980648628eae548d2: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:55.907719  4650 rebalancer_tool.cc:1136] tablet 190ebb1335d64b279a41fa38feff22b4: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:15:55.985129  4650 rebalancer_tool.cc:1136] tablet 51224b6355ed41dbb0703b7feb73c223: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:56.026535  4650 rebalancer_tool.cc:1136] tablet f0a088d230d0462980648628eae548d2: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:56.204140  4650 rebalancer_tool.cc:1136] tablet 43e248f0cb764d229802851c4e73cc96: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
I20240503 02:15:56.400085  4650 rebalancer_tool.cc:1136] tablet 9253de3f798f4a208d566765ed2ba54d: '6d54feb0bbc14c109783371d2939b234' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:56.520687  4650 rebalancer_tool.cc:1136] tablet 0137e646f1bd452b950d870107afa060: '08055479ea104aa290d19955d3720850' -> '408af0ab5520465e942cc714bfc6c73b' move completed: OK
I20240503 02:15:56.621155  4650 rebalancer_tool.cc:1136] tablet a7ab8b0c8db54565a6066b9a7d1569d8: '08bef4c144bc4e4d8cb5836aee8da852' -> '7fd07044862a481aa387698017467c01' move completed: OK
I20240503 02:15:56.796268  4650 rebalancer_tool.cc:1136] tablet 586ebf467f0f427fa62dee6223a39848: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:57.076161  4650 rebalancer_tool.cc:1136] tablet 62a3dcc2de594c8982477a0229f71191: 'e9fdaf326bb64edabc43b366df870f83' -> '' move completed: OK
I20240503 02:15:57.224974  4650 rebalancer_tool.cc:1136] tablet 9b822c691d824b2ba25e967535c9734e: '08bef4c144bc4e4d8cb5836aee8da852' -> '9e699d9f984b4cf2b8b468322759b78c' move completed: OK
I20240503 02:15:57.378052  4650 rebalancer_tool.cc:1074] tablet 8870feaf47a64a929dd2e584ece47218: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:57.511646  4650 rebalancer_tool.cc:1074] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:57.724264  4650 rebalancer_tool.cc:1074] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:15:57.909611  4650 rebalancer_tool.cc:1074] tablet 771a4190142b4fd5a9479417122c0881: '6d54feb0bbc14c109783371d2939b234' -> '' move scheduled
I20240503 02:15:58.032918  4650 rebalancer_tool.cc:1074] tablet ae239d73f0bc4d9aac025e40489e81a6: '08055479ea104aa290d19955d3720850' -> '' move scheduled
I20240503 02:15:58.139364  4650 rebalancer_tool.cc:1136] tablet ae239d73f0bc4d9aac025e40489e81a6: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
W20240503 02:15:59.796093 11152 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.67:38127, user_credentials={real_user=slave}} blocked reactor thread for 49528.2us
I20240503 02:16:02.494787  4650 rebalancer_tool.cc:1136] tablet 1eeb69e5eb3c459ca749aa37672ec057: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:07.956073  4650 rebalancer_tool.cc:1074] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:08.085323  4650 rebalancer_tool.cc:1074] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:08.270212  4650 rebalancer_tool.cc:1136] tablet a6e0d39087944e4ea13a17ca4af478ab: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:08.345053  4650 rebalancer_tool.cc:1136] tablet 0e61eb7d9e6041b6b99d5d5328943140: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:10.007365  4650 rebalancer_tool.cc:1136] tablet 51dbf0fd54b14c2fa21d3e62104c9b87: '08055479ea104aa290d19955d3720850' -> '' move completed: OK
I20240503 02:16:13.023375  4650 rebalancer_tool.cc:1074] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move scheduled
I20240503 02:16:13.125792  4650 rebalancer_tool.cc:1136] tablet f084f1783d4d477e9c3791984e681f4c: '08bef4c144bc4e4d8cb5836aee8da852' -> '' move completed: OK
I20240503 02:16:16.837489  4650 rebalancer_tool.cc:1136] tablet 771a4190142b4fd5a9479417122c0881: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
W20240503 02:18:43.565418 32210 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 85478.9us
W20240503 02:20:28.657423 14006 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.218.65:34489, user_credentials={real_user=slave}} blocked reactor thread for 45151.1us
I20240503 02:21:09.656996  4650 rebalancer_tool.cc:1136] tablet 8870feaf47a64a929dd2e584ece47218: '6d54feb0bbc14c109783371d2939b234' -> '' move completed: OK
Illegal state: tablet server 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): unacceptable health status UNAVAILABLE

E20240503 02:21:10.474503 21353 external_mini_cluster.cc:943] Process with UUID 7fd07044862a481aa387698017467c01 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240503 02:21:10.476238 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2199
I20240503 02:21:10.483883  2321 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:10.851588 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2199
W20240503 02:21:10.930400  4129 connection.cc:535] server connection from 127.20.218.65:34381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930414  4395 connection.cc:535] server connection from 127.20.218.65:43879 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930509  2478 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.930666  2478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 26 similar messages]
W20240503 02:21:10.930968  4263 connection.cc:535] server connection from 127.20.218.65:42755 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:10.931151 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2330
W20240503 02:21:10.931264  2348 connection.cc:535] server connection from 127.20.218.65:33571 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931377  2606 connection.cc:535] server connection from 127.20.218.65:58531 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931449  2348 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931547  2348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 16 similar messages]
W20240503 02:21:10.931592  2609 connection.cc:535] client connection to 127.20.218.65:34489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.931669  2609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 31 similar messages]
W20240503 02:21:10.932622  2124 connection.cc:535] server connection from 127.20.218.65:35201 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:10.945475  2478 consensus_peers.cc:485] T 9253de3f798f4a208d566765ed2ba54d P 08bef4c144bc4e4d8cb5836aee8da852 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946141  2348 consensus_peers.cc:485] T eadcb49d12664e6fb5a5ad8dbe47f342 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946278  2348 consensus_peers.cc:485] T a7ab8b0c8db54565a6066b9a7d1569d8 P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946297  2609 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:10.946388  2348 consensus_peers.cc:485] T a6e0d39087944e4ea13a17ca4af478ab P 6d54feb0bbc14c109783371d2939b234 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:10.952958  2451 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:11.326215 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2330
W20240503 02:21:11.385838  4132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.386596  4129 connection.cc:535] server connection from 127.20.218.66:32795 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:11.387169  2607 connection.cc:535] server connection from 127.20.218.66:53211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:11.387777 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2460
W20240503 02:21:11.387827  4132 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Couldn't send request to peer 08055479ea104aa290d19955d3720850. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.388332  4129 consensus_peers.cc:485] T 51224b6355ed41dbb0703b7feb73c223 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.398114  2577 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:21:11.415127  2606 consensus_peers.cc:485] T 0e61eb7d9e6041b6b99d5d5328943140 P e9fdaf326bb64edabc43b366df870f83 -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.428690  4260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240503 02:21:11.429900  4260 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Couldn't send request to peer 6d54feb0bbc14c109783371d2939b234. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.779556 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2460
I20240503 02:21:11.831924 19041 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.832151 19041 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.833019 19041 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:11.833812  4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "408af0ab5520465e942cc714bfc6c73b" is_pre_election: true
I20240503 02:21:11.833817  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "e9fdaf326bb64edabc43b366df870f83" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:11.834766  2608 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e9fdaf326bb64edabc43b366df870f83; no voters: 1d36807868fe4a82bca42c25831ce8a2, 408af0ab5520465e942cc714bfc6c73b
I20240503 02:21:11.835180 19041 raft_consensus.cc:2726] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:11.835716  2609 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P e9fdaf326bb64edabc43b366df870f83 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.858322  4262 connection.cc:535] server connection from 127.20.218.67:56343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:11.858935 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2591
W20240503 02:21:11.859361  2607 connection.cc:535] client connection to 127.20.218.67:38127 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:11.860270  4261 consensus_peers.cc:485] T 592cf4d5b35e49bfa4173343d64367f4 P 9e699d9f984b4cf2b8b468322759b78c -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.860800  2607 consensus_peers.cc:485] T 1eeb69e5eb3c459ca749aa37672ec057 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.860975  2607 consensus_peers.cc:485] T 190ebb1335d64b279a41fa38feff22b4 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.861092  2607 consensus_peers.cc:485] T 43e248f0cb764d229802851c4e73cc96 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:11.861204  2607 consensus_peers.cc:485] T a1949a5c7cc741b19da021eccb7d0dd1 P e9fdaf326bb64edabc43b366df870f83 -> Peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Couldn't send request to peer 08bef4c144bc4e4d8cb5836aee8da852. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:11.865859  2712 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:11.916859 19310 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.917119 19310 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.918946 19310 leader_election.cc:288] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:11.919937  4397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:11.922715  4397 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:11.926512  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a62b3ef44e94e92b0feed9c228f5702" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:11.927214  4396 leader_election.cc:302] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:21:11.927567 19310 raft_consensus.cc:2726] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:11.982729 18732 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:11.982975 18732 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:11.983757 18732 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:11.984827  4461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "1d36807868fe4a82bca42c25831ce8a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "408af0ab5520465e942cc714bfc6c73b" is_pre_election: true
W20240503 02:21:11.985209  4132 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:11.985525  4132 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2; no voters: 08055479ea104aa290d19955d3720850, 408af0ab5520465e942cc714bfc6c73b
I20240503 02:21:11.985805 18732 raft_consensus.cc:2726] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.239439 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2591
I20240503 02:21:12.244143 16912 raft_consensus.cc:482] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.244369 16912 raft_consensus.cc:504] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:12.245642 16912 leader_election.cc:288] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.248453  4260 leader_election.cc:334] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:12.252852  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b822c691d824b2ba25e967535c9734e" candidate_uuid: "9e699d9f984b4cf2b8b468322759b78c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:12.253546  4262 leader_election.cc:302] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.253839 16912 raft_consensus.cc:2726] T 9b822c691d824b2ba25e967535c9734e P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:12.301470  4263 connection.cc:535] server connection from 127.20.218.68:41785 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:12.302199 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4114
W20240503 02:21:12.302376 19312 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:21:11.919131 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.218.68:38103 (local address 127.20.218.71:33021)
0503 02:21:11.919776 (+   645us) negotiation.cc:107] Waiting for socket to connect
0503 02:21:11.920655 (+   879us) client_negotiation.cc:174] Beginning negotiation
0503 02:21:11.920757 (+   102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:21:12.301665 (+380908us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":475,"threads_started":1}
W20240503 02:21:12.302896  4131 consensus_peers.cc:485] T 771a4190142b4fd5a9479417122c0881 P 1d36807868fe4a82bca42c25831ce8a2 -> Peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Couldn't send request to peer e9fdaf326bb64edabc43b366df870f83. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:21:12.303803  4396 leader_election.cc:334] T 7a62b3ef44e94e92b0feed9c228f5702 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:12.308266  4232 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:12.355073 16912 raft_consensus.cc:482] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.355185 19310 raft_consensus.cc:482] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.355333 19310 raft_consensus.cc:504] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } }
I20240503 02:21:12.355309 16912 raft_consensus.cc:504] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:12.356127 16912 leader_election.cc:288] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:12.356158 19310 leader_election.cc:288] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:12.356304 19310 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.356451 19310 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.357074 19310 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.358361  4395 leader_election.cc:334] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.358772  4260 leader_election.cc:334] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.358889  4263 leader_election.cc:334] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:12.358969  4397 leader_election.cc:334] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.359007  4263 leader_election.cc:302] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.359092  4397 leader_election.cc:302] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:12.359285 16912 raft_consensus.cc:2726] T 586ebf467f0f427fa62dee6223a39848 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.359329 19310 raft_consensus.cc:2726] T 02e0b954008c4f30af414da6cb5d59c2 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:21:12.359575  4397 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.406826 16912 raft_consensus.cc:482] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.407052 16912 raft_consensus.cc:504] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } }
I20240503 02:21:12.407786 16912 leader_election.cc:288] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.409189  4260 leader_election.cc:334] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.409348  4263 leader_election.cc:334] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.409582  4263 leader_election.cc:302] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.409893 16912 raft_consensus.cc:2726] T eadcb49d12664e6fb5a5ad8dbe47f342 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.410866 19310 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.411054 19310 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.412015 19310 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.413246  4394 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.413753  4396 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:12.413941  4396 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:12.414227 19310 raft_consensus.cc:2726] T f084f1783d4d477e9c3791984e681f4c P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.419021  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f084f1783d4d477e9c3791984e681f4c" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:12.470412 19310 raft_consensus.cc:482] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:12.470688 19310 raft_consensus.cc:504] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.471503 19310 leader_election.cc:288] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.472844  4395 leader_election.cc:334] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.472997  4397 leader_election.cc:334] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.473119  4397 leader_election.cc:302] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 08bef4c144bc4e4d8cb5836aee8da852
I20240503 02:21:12.473393 19310 raft_consensus.cc:2726] T 9253de3f798f4a208d566765ed2ba54d P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.579079 19310 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:12.579313 19310 raft_consensus.cc:504] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.580107 19310 leader_election.cc:288] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
I20240503 02:21:12.581068  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8870feaf47a64a929dd2e584ece47218" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
W20240503 02:21:12.581676  4395 leader_election.cc:334] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
I20240503 02:21:12.581818  4395 leader_election.cc:302] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 9e699d9f984b4cf2b8b468322759b78c
I20240503 02:21:12.582108 19310 raft_consensus.cc:2726] T 8870feaf47a64a929dd2e584ece47218 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:12.653432  4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "734027e195414b15811ac9d32b132db8" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1d36807868fe4a82bca42c25831ce8a2" is_pre_election: true
I20240503 02:21:12.654162 18732 raft_consensus.cc:482] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.654330 18732 raft_consensus.cc:504] T 7a62b3ef44e94e92b0feed9c228f5702 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.654474  4194 raft_consensus.cc:2454] T 734027e195414b15811ac9d32b132db8 P 1d36807868fe4a82bca42c25831ce8a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408af0ab5520465e942cc714bfc6c73b in term 1.
I20240503 02:21:12.655009 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4114
I20240503 02:21:12.656944  4396 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d36807868fe4a82bca42c25831ce8a2, 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850
I20240503 02:21:12.657279 19310 raft_consensus.cc:2781] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:21:12.657444 19310 raft_consensus.cc:482] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 08055479ea104aa290d19955d3720850)
I20240503 02:21:12.657537 19310 raft_consensus.cc:3037] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:12.659847 19310 raft_consensus.cc:504] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.660475 19310 leader_election.cc:288] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: Requested vote from peers 08055479ea104aa290d19955d3720850 (127.20.218.65:34489), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
W20240503 02:21:12.661865  4397 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
W20240503 02:21:12.712978  4396 connection.cc:535] client connection to 127.20.218.69:39741 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:12.713310 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4245
W20240503 02:21:12.715031  4396 leader_election.cc:334] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Network error: Client connection negotiation failed: client connection to 127.20.218.69:39741: connect: Connection refused (error 111)
I20240503 02:21:12.715152  4396 leader_election.cc:302] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 1d36807868fe4a82bca42c25831ce8a2
I20240503 02:21:12.715456 19310 raft_consensus.cc:2726] T 734027e195414b15811ac9d32b132db8 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.719637  4362 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:12.824241 19310 raft_consensus.cc:482] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.824481 19310 raft_consensus.cc:504] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "1d36807868fe4a82bca42c25831ce8a2" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 39741 } }
I20240503 02:21:12.824838 19322 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.825004 19322 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.825305 19310 leader_election.cc:288] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741)
I20240503 02:21:12.825824 19322 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.827643  4395 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:12.828003  4394 leader_election.cc:334] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.828121  4394 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:12.828222  4394 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
W20240503 02:21:12.828410  4396 leader_election.cc:334] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d36807868fe4a82bca42c25831ce8a2 (127.20.218.69:39741): Network error: Client connection negotiation failed: client connection to 127.20.218.69:39741: connect: Connection refused (error 111)
I20240503 02:21:12.828521  4396 leader_election.cc:302] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 1d36807868fe4a82bca42c25831ce8a2, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.828644 19322 raft_consensus.cc:2726] T 75ba33132e784f408f3e26882aea3d15 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:12.828753 19310 raft_consensus.cc:2726] T 9f7c9c29384949fb95cb0c3b765f57b0 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.828994 19310 raft_consensus.cc:482] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:12.829157 19310 raft_consensus.cc:504] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "08055479ea104aa290d19955d3720850" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 34489 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:12.829726 19310 leader_election.cc:288] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 08055479ea104aa290d19955d3720850 (127.20.218.65:34489)
W20240503 02:21:12.831068  4394 leader_election.cc:334] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:12.831239  4397 leader_election.cc:334] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08055479ea104aa290d19955d3720850 (127.20.218.65:34489): Network error: Client connection negotiation failed: client connection to 127.20.218.65:34489: connect: Connection refused (error 111)
I20240503 02:21:12.831357  4397 leader_election.cc:302] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 08055479ea104aa290d19955d3720850, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:12.831614 19310 raft_consensus.cc:2726] T a6e0d39087944e4ea13a17ca4af478ab P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:12.927016 19310 raft_consensus.cc:482] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e9fdaf326bb64edabc43b366df870f83)
I20240503 02:21:12.927244 19310 raft_consensus.cc:504] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:12.927958 19310 leader_election.cc:288] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 9e699d9f984b4cf2b8b468322759b78c (127.20.218.70:33893)
W20240503 02:21:12.929282  4396 leader_election.cc:334] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:13.077709 16912 raft_consensus.cc:482] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:13.078670  4326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ba33132e784f408f3e26882aea3d15" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:13.078795  4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0137e646f1bd452b950d870107afa060" candidate_uuid: "408af0ab5520465e942cc714bfc6c73b" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "9e699d9f984b4cf2b8b468322759b78c" is_pre_election: true
I20240503 02:21:13.078934  4326 raft_consensus.cc:2454] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408af0ab5520465e942cc714bfc6c73b in term 2.
I20240503 02:21:13.079094 16912 raft_consensus.cc:504] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:13.079701  4394 leader_election.cc:302] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 408af0ab5520465e942cc714bfc6c73b; no voters: 9e699d9f984b4cf2b8b468322759b78c, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:13.080456 19310 raft_consensus.cc:2726] T 0137e646f1bd452b950d870107afa060 P 408af0ab5520465e942cc714bfc6c73b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:21:13.080780 16912 leader_election.cc:288] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
I20240503 02:21:13.080915 16912 raft_consensus.cc:482] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:13.081045 16912 raft_consensus.cc:504] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } }
I20240503 02:21:13.081691 16912 leader_election.cc:288] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903)
I20240503 02:21:13.081836 16912 raft_consensus.cc:482] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d54feb0bbc14c109783371d2939b234)
I20240503 02:21:13.081984 16912 raft_consensus.cc:504] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "6d54feb0bbc14c109783371d2939b234" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34903 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } } peers { permanent_uuid: "9e699d9f984b4cf2b8b468322759b78c" member_type: VOTER last_known_addr { host: "127.20.218.70" port: 33893 } }
I20240503 02:21:13.083647 16912 leader_election.cc:288] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127), 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903), 408af0ab5520465e942cc714bfc6c73b (127.20.218.71:38285)
W20240503 02:21:13.084133  4261 leader_election.cc:334] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:13.085044  4260 leader_election.cc:334] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:13.085249  4260 leader_election.cc:302] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
W20240503 02:21:13.085795  4261 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127): Network error: Client connection negotiation failed: client connection to 127.20.218.67:38127: connect: Connection refused (error 111)
W20240503 02:21:13.085842  4260 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
W20240503 02:21:13.086058  4260 leader_election.cc:334] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d54feb0bbc14c109783371d2939b234 (127.20.218.66:34903): Network error: Client connection negotiation failed: client connection to 127.20.218.66:34903: connect: Connection refused (error 111)
I20240503 02:21:13.086194  4260 leader_election.cc:302] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 08bef4c144bc4e4d8cb5836aee8da852, 6d54feb0bbc14c109783371d2939b234
I20240503 02:21:13.086490 19330 raft_consensus.cc:2726] T 51dbf0fd54b14c2fa21d3e62104c9b87 P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:13.086632 16912 raft_consensus.cc:2726] T 75ba33132e784f408f3e26882aea3d15 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:21:13.089545  4262 leader_election.cc:334] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103): Network error: Client connection negotiation failed: client connection to 127.20.218.68:38103: connect: Connection refused (error 111)
I20240503 02:21:13.089736  4262 leader_election.cc:302] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 9e699d9f984b4cf2b8b468322759b78c; no voters: 6d54feb0bbc14c109783371d2939b234, e9fdaf326bb64edabc43b366df870f83
I20240503 02:21:13.090090 16912 raft_consensus.cc:2726] T f084f1783d4d477e9c3791984e681f4c P 9e699d9f984b4cf2b8b468322759b78c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:21:13.090063 19328 raft_consensus.cc:482] T 8870feaf47a64a929dd2e584ece47218 P 9e699d9f984b4cf2b8b468322759b78c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 08bef4c144bc4e4d8cb5836aee8da852)
I20240503 02:21:13.090596 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4245
W20240503 02:21:13.150774 19332 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:21:13.091666 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.218.70:40555 (local address 127.20.218.71:38285)
0503 02:21:13.092160 (+   494us) server_negotiation.cc:206] Beginning negotiation
0503 02:21:13.092169 (+     9us) server_negotiation.cc:399] Waiting for connection header
0503 02:21:13.150487 (+ 58318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.218.70:40555: BlockingRecv error: recv got EOF from 127.20.218.70:40555 (error 108)
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":228,"threads_started":1}
I20240503 02:21:13.150828 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 4377
I20240503 02:21:13.156821  4497 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:21:13.392470  2126 catalog_manager.cc:4573] TS 7fd07044862a481aa387698017467c01 (127.20.218.72:37033): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 51dbf0fd54b14c2fa21d3e62104c9b87: Network error: Client connection negotiation failed: client connection to 127.20.218.72:37033: connect: Connection refused (error 111)
I20240503 02:21:13.485200 19310 raft_consensus.cc:482] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e9fdaf326bb64edabc43b366df870f83)
I20240503 02:21:13.485471 19310 raft_consensus.cc:504] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9fdaf326bb64edabc43b366df870f83" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 38103 } } peers { permanent_uuid: "08bef4c144bc4e4d8cb5836aee8da852" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 38127 } } peers { permanent_uuid: "408af0ab5520465e942cc714bfc6c73b" member_type: VOTER last_known_addr { host: "127.20.218.71" port: 38285 } }
I20240503 02:21:13.486428 19310 leader_election.cc:288] T 190ebb1335d64b279a41fa38feff22b4 P 408af0ab5520465e942cc714bfc6c73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9fdaf326bb64edabc43b366df870f83 (127.20.218.68:38103), 08bef4c144bc4e4d8cb5836aee8da852 (127.20.218.67:38127)
I20240503 02:21:13.487200 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 4377
I20240503 02:21:13.544466 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 2109
I20240503 02:21:13.550570  2168 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:13.899665 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 2109
2024-05-03T02:21:13Z chronyd exiting
I20240503 02:21:13.969877 21353 test_util.cc:175] -----------------------------------------------
I20240503 02:21:13.970005 21353 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714702283380610-21353-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (384882 ms)
[----------] 1 test from ConcurrentRebalancersTest (384882 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240503 02:21:13.974650 21353 test_util.cc:268] Using random seed: 1605556047
I20240503 02:21:14.017452 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:21:14.017537 21353 ts_itest-base.cc:113] --------------
I20240503 02:21:14.017560 21353 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:21:14.017596 21353 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:21:14.017637 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:21:14Z Disabled control of system clock
I20240503 02:21:14.077446 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:46685
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:46685
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240503 02:21:14.468412 19347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:14.468811 19347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:14.468966 19347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:14.477020 19347 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:14.477202 19347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:21:14.477293 19347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:14.477401 19347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:21:14.477469 19347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:21:14.487061 19347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:46685
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:46685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:14.488525 19347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:14.490407 19347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:14.501693 19352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:14.501777 19355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:14.501726 19353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:14.502664 19347 server_base.cc:1034] running on GCE node
I20240503 02:21:14.503490 19347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:14.505039 19347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:14.506264 19347 hybrid_clock.cc:648] HybridClock initialized: now 1714702874506196 us; error 54 us; skew 500 ppm
I20240503 02:21:14.506647 19347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:14.509155 19347 webserver.cc:460] Webserver started at http://127.20.218.126:41019/ using document root <none> and password file <none>
I20240503 02:21:14.509953 19347 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:14.510092 19347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:14.510428 19347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:14.513196 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5720469d90d84bbe82d6b2d18f5589d4"
format_stamp: "Formatted at 2024-05-03 02:21:14 on dist-test-slave-81lq"
I20240503 02:21:14.514104 19347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5720469d90d84bbe82d6b2d18f5589d4"
format_stamp: "Formatted at 2024-05-03 02:21:14 on dist-test-slave-81lq"
I20240503 02:21:14.519623 19347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:21:14.523262 19361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:14.524660 19347 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:14.524845 19347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5720469d90d84bbe82d6b2d18f5589d4"
format_stamp: "Formatted at 2024-05-03 02:21:14 on dist-test-slave-81lq"
I20240503 02:21:14.525084 19347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:14.570911 19347 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:14.571853 19347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:14.572182 19347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:14.591388 19347 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:46685
I20240503 02:21:14.591411 19412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:46685 every 8 connection(s)
I20240503 02:21:14.592775 19347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:21:14.594239 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 19347
I20240503 02:21:14.594481 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:21:14.597676 19413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:14.608727 19413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4: Bootstrap starting.
I20240503 02:21:14.611836 19413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:14.612905 19413 log.cc:824] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:14.615847 19413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4: No bootstrap required, opened a new log
I20240503 02:21:14.624044 19413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } }
I20240503 02:21:14.624431 19413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:14.624550 19413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5720469d90d84bbe82d6b2d18f5589d4, State: Initialized, Role: FOLLOWER
I20240503 02:21:14.625187 19413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } }
I20240503 02:21:14.625420 19413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:14.625530 19413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:14.625660 19413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:14.627771 19413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } }
I20240503 02:21:14.628227 19413 leader_election.cc:302] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5720469d90d84bbe82d6b2d18f5589d4; no voters: 
I20240503 02:21:14.628816 19413 leader_election.cc:288] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:14.628947 19418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:14.629626 19418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5720469d90d84bbe82d6b2d18f5589d4, State: Running, Role: LEADER
I20240503 02:21:14.630215 19418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } }
I20240503 02:21:14.630916 19413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:21:14.633230 19419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5720469d90d84bbe82d6b2d18f5589d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } } }
I20240503 02:21:14.633780 19419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:14.634327 19420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5720469d90d84bbe82d6b2d18f5589d4. Latest consensus state: current_term: 1 leader_uuid: "5720469d90d84bbe82d6b2d18f5589d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5720469d90d84bbe82d6b2d18f5589d4" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 46685 } } }
I20240503 02:21:14.634644 19420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4 [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:14.635243 19425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:21:14.641443 19425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:21:14.650274 19425 catalog_manager.cc:1347] Generated new cluster ID: daa0b3def7674197a5f01e6c5325fe70
I20240503 02:21:14.650404 19425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:21:14.682330 19425 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:21:14.684135 19425 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:21:14.711045 19425 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5720469d90d84bbe82d6b2d18f5589d4: Generated new TSK 0
I20240503 02:21:14.712208 19425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:21:14.722580 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:21:15.077971 19437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:15.078313 19437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:15.078456 19437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:15.086310 19437 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:15.086483 19437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:15.086652 19437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:21:15.096242 19437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:15.097785 19437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:15.099628 19437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:15.112674 19445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:15.112591 19443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:15.112632 19442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:15.113518 19437 server_base.cc:1034] running on GCE node
I20240503 02:21:15.114181 19437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:15.115177 19437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:15.116379 19437 hybrid_clock.cc:648] HybridClock initialized: now 1714702875116319 us; error 59 us; skew 500 ppm
I20240503 02:21:15.116672 19437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:15.119181 19437 webserver.cc:460] Webserver started at http://127.20.218.65:32823/ using document root <none> and password file <none>
I20240503 02:21:15.119997 19437 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:15.120136 19437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:15.120468 19437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:15.123236 19437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e0e4fb3769f34dfda47462f0550e8e67"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.124080 19437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e0e4fb3769f34dfda47462f0550e8e67"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.129799 19437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.004s
I20240503 02:21:15.133663 19451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.135257 19437 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:21:15.135459 19437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e0e4fb3769f34dfda47462f0550e8e67"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.135689 19437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:15.150835 19437 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:15.151716 19437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:15.151995 19437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:15.153020 19437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:15.154594 19437 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:15.154692 19437 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.154801 19437 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:15.154872 19437 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.187249 19437 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:38371
I20240503 02:21:15.187310 19563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:38371 every 8 connection(s)
I20240503 02:21:15.188601 19437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:21:15.198019 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 19437
I20240503 02:21:15.198292 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:21:15.202109 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:21:15.202221 19564 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:15.202553 19564 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:15.203316 19564 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:15.205449 19378 ts_manager.cc:194] Registered new tserver with Master: e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65:38371)
I20240503 02:21:15.207037 19378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:33151
W20240503 02:21:15.554219 19568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:15.554533 19568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:15.554638 19568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:15.562297 19568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:15.562443 19568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:15.562562 19568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:21:15.571892 19568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:15.573348 19568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:15.575162 19568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:15.587529 19573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:15.587529 19574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:15.587802 19576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:15.588518 19568 server_base.cc:1034] running on GCE node
I20240503 02:21:15.589109 19568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:15.590098 19568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:15.591338 19568 hybrid_clock.cc:648] HybridClock initialized: now 1714702875591272 us; error 44 us; skew 500 ppm
I20240503 02:21:15.591755 19568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:15.594712 19568 webserver.cc:460] Webserver started at http://127.20.218.66:38135/ using document root <none> and password file <none>
I20240503 02:21:15.595665 19568 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:15.595803 19568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:15.596176 19568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:15.599789 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.600857 19568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.607860 19568 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.005s	sys 0.004s
I20240503 02:21:15.612469 19582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.613967 19568 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:21:15.614162 19568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50"
format_stamp: "Formatted at 2024-05-03 02:21:15 on dist-test-slave-81lq"
I20240503 02:21:15.614372 19568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:15.644012 19568 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:15.644913 19568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:15.645203 19568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:15.646317 19568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:15.647853 19568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:15.647945 19568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.648065 19568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:15.648147 19568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:15.682047 19568 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:40887
I20240503 02:21:15.682112 19694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:40887 every 8 connection(s)
I20240503 02:21:15.683411 19568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:21:15.687700 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 19568
I20240503 02:21:15.687934 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:21:15.691922 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:21:15.698163 19695 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:15.698525 19695 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:15.699486 19695 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:15.701073 19378 ts_manager.cc:194] Registered new tserver with Master: b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66:40887)
I20240503 02:21:15.701942 19378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:34267
W20240503 02:21:16.048827 19698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:16.049154 19698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:16.049247 19698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:16.057158 19698 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:16.057309 19698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:16.057543 19698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:21:16.067279 19698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:16.068689 19698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:16.070555 19698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:16.083426 19703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:16.083456 19704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:16.083427 19706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:16.084338 19698 server_base.cc:1034] running on GCE node
I20240503 02:21:16.084941 19698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:16.086092 19698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:16.087314 19698 hybrid_clock.cc:648] HybridClock initialized: now 1714702876087254 us; error 62 us; skew 500 ppm
I20240503 02:21:16.087672 19698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:16.090266 19698 webserver.cc:460] Webserver started at http://127.20.218.67:39191/ using document root <none> and password file <none>
I20240503 02:21:16.091100 19698 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:16.091228 19698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:16.091575 19698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:16.094398 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "67ed198a70f54a919d2be9d42980ef7b"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.095274 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "67ed198a70f54a919d2be9d42980ef7b"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.104619 19698 fs_manager.cc:691] Time spent creating directory manager: real 0.009s	user 0.006s	sys 0.001s
I20240503 02:21:16.108556 19712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.110149 19698 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:16.110339 19698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "67ed198a70f54a919d2be9d42980ef7b"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.110571 19698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:16.133208 19698 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:16.134195 19698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:16.134495 19698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:16.135552 19698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:16.137224 19698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:16.137352 19698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.137485 19698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:16.137566 19698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.172394 19698 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:35137
I20240503 02:21:16.172448 19824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:35137 every 8 connection(s)
I20240503 02:21:16.173760 19698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:21:16.180639 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 19698
I20240503 02:21:16.180903 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:21:16.184166 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:21:16.187546 19825 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:16.191018 19825 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:16.192108 19825 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:16.193845 19378 ts_manager.cc:194] Registered new tserver with Master: 67ed198a70f54a919d2be9d42980ef7b (127.20.218.67:35137)
I20240503 02:21:16.194703 19378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:41099
I20240503 02:21:16.210068 19564 heartbeater.cc:503] Master 127.20.218.126:46685 was elected leader, sending a full tablet report...
W20240503 02:21:16.546816 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:16.547156 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:16.547289 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:16.555235 19828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:16.555413 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:16.555591 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:16.565002 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:16.566480 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:16.568361 19828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:16.581290 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:16.581264 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:16.581239 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:16.582234 19828 server_base.cc:1034] running on GCE node
I20240503 02:21:16.582895 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:16.583923 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:16.585142 19828 hybrid_clock.cc:648] HybridClock initialized: now 1714702876585091 us; error 45 us; skew 500 ppm
I20240503 02:21:16.585512 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:16.591863 19828 webserver.cc:460] Webserver started at http://127.20.218.68:39503/ using document root <none> and password file <none>
I20240503 02:21:16.592689 19828 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:16.592829 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:16.593149 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:16.595899 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "95827103554543fd829afd3e8644de1d"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.596729 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "95827103554543fd829afd3e8644de1d"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.602326 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240503 02:21:16.605911 19842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.607273 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240503 02:21:16.607447 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "95827103554543fd829afd3e8644de1d"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:16.607678 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:16.627171 19828 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:16.628080 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:16.628362 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:16.629390 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:16.630931 19828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:16.631047 19828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.631162 19828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:16.631237 19828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:16.665242 19828 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:34269
I20240503 02:21:16.665401 19954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:34269 every 8 connection(s)
I20240503 02:21:16.666635 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:16.668221 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 19828
I20240503 02:21:16.668402 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:21:16.679698 19955 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:16.679993 19955 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:16.680742 19955 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:16.682185 19378 ts_manager.cc:194] Registered new tserver with Master: 95827103554543fd829afd3e8644de1d (127.20.218.68:34269)
I20240503 02:21:16.683315 19378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:56107
I20240503 02:21:16.684415 21353 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:21:16.705076 19695 heartbeater.cc:503] Master 127.20.218.126:46685 was elected leader, sending a full tablet report...
I20240503 02:21:16.701526 19378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36274:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240503 02:21:16.730278 19630 tablet_service.cc:1467] Processing CreateTablet for tablet 9dac7d8b040d45fbbda6596176a40e3d (DEFAULT_TABLE table=TestTable [id=899eeee8971b483ea5e60f3b7c276bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:16.731721 19630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dac7d8b040d45fbbda6596176a40e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:16.742249 19969 tablet_bootstrap.cc:492] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:16.745450 19969 tablet_bootstrap.cc:654] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:16.746600 19969 log.cc:824] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:16.749119 19969 tablet_bootstrap.cc:492] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:16.749400 19969 ts_tablet_manager.cc:1367] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240503 02:21:16.756093 19969 raft_consensus.cc:348] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:16.756443 19969 raft_consensus.cc:374] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:16.756511 19969 raft_consensus.cc:729] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:16.757124 19969 consensus_queue.cc:260] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:16.757315 19969 raft_consensus.cc:388] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:16.757508 19969 raft_consensus.cc:482] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:16.757654 19969 raft_consensus.cc:3037] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:16.759625 19969 raft_consensus.cc:504] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:16.760191 19969 leader_election.cc:302] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:16.760960 19969 leader_election.cc:288] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:16.761096 19971 raft_consensus.cc:2781] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:16.761731 19971 raft_consensus.cc:686] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:16.762367 19971 consensus_queue.cc:237] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:16.763336 19969 ts_tablet_manager.cc:1398] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.008s
I20240503 02:21:16.767443 19378 catalog_manager.cc:5484] T 9dac7d8b040d45fbbda6596176a40e3d P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:16.777393 21353 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:21:16.779487 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e0e4fb3769f34dfda47462f0550e8e67 to finish bootstrapping
I20240503 02:21:16.788920 21353 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b1ee1626ca1d4a52a2dda4ba42b8ce50 to finish bootstrapping
I20240503 02:21:16.797053 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 67ed198a70f54a919d2be9d42980ef7b to finish bootstrapping
I20240503 02:21:16.805682 21353 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 95827103554543fd829afd3e8644de1d to finish bootstrapping
I20240503 02:21:16.814805 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 19698
I20240503 02:21:16.815860 19820 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:16.897416 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 19698
I20240503 02:21:16.915410 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 19828
I20240503 02:21:16.916565 19950 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:16.999284 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 19828
I20240503 02:21:20.769069 19377 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36274:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:21:20.789557 19497 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd107a6b54646e6b93e03959c7a1637 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af679ede4554c5b95a935244331d107]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:20.789469 19498 tablet_service.cc:1467] Processing CreateTablet for tablet ea65793ff1dc42fdb30fdf6bf2b12252 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af679ede4554c5b95a935244331d107]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:20.789516 19499 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8511a8bb1b4f2abe028021a8aac250 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af679ede4554c5b95a935244331d107]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:20.791034 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd107a6b54646e6b93e03959c7a1637. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:20.791499 19499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8511a8bb1b4f2abe028021a8aac250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:20.791729 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea65793ff1dc42fdb30fdf6bf2b12252. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:20.803439 19988 tablet_bootstrap.cc:492] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:20.806561 19988 tablet_bootstrap.cc:654] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:20.807616 19988 log.cc:824] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:20.810199 19988 tablet_bootstrap.cc:492] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:20.810459 19988 ts_tablet_manager.cc:1367] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.001s
I20240503 02:21:20.816709 19988 raft_consensus.cc:348] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.817101 19988 raft_consensus.cc:374] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:20.817229 19988 raft_consensus.cc:729] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:20.817868 19988 consensus_queue.cc:260] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.818068 19988 raft_consensus.cc:388] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:20.818205 19988 raft_consensus.cc:482] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:20.818352 19988 raft_consensus.cc:3037] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:20.820613 19988 raft_consensus.cc:504] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.821108 19988 leader_election.cc:302] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:20.821791 19988 leader_election.cc:288] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:20.821939 19990 raft_consensus.cc:2781] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:20.822746 19990 raft_consensus.cc:686] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:20.823312 19990 consensus_queue.cc:237] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.823866 19988 ts_tablet_manager.cc:1398] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.003s
I20240503 02:21:20.824231 19988 tablet_bootstrap.cc:492] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:20.826171 19988 tablet_bootstrap.cc:654] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:20.828047 19377 catalog_manager.cc:5484] T 5bd107a6b54646e6b93e03959c7a1637 P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:20.834484 19988 tablet_bootstrap.cc:492] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:20.834682 19988 ts_tablet_manager.cc:1367] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20240503 02:21:20.835534 19988 raft_consensus.cc:348] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.835686 19988 raft_consensus.cc:374] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:20.835755 19988 raft_consensus.cc:729] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:20.835953 19988 consensus_queue.cc:260] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.836099 19988 raft_consensus.cc:388] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:20.836146 19988 raft_consensus.cc:482] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:20.836220 19988 raft_consensus.cc:3037] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:20.838160 19988 raft_consensus.cc:504] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.838377 19988 leader_election.cc:302] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:20.838541 19988 leader_election.cc:288] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:20.838635 19990 raft_consensus.cc:2781] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:20.839056 19988 ts_tablet_manager.cc:1398] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:21:20.839118 19990 raft_consensus.cc:686] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:20.839397 19988 tablet_bootstrap.cc:492] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:20.839397 19990 consensus_queue.cc:237] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.841300 19988 tablet_bootstrap.cc:654] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:20.841972 19377 catalog_manager.cc:5484] T ea65793ff1dc42fdb30fdf6bf2b12252 P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:20.843068 19988 tablet_bootstrap.cc:492] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:20.843225 19988 ts_tablet_manager.cc:1367] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:21:20.843775 19988 raft_consensus.cc:348] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.843981 19988 raft_consensus.cc:374] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:20.844070 19988 raft_consensus.cc:729] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:20.844331 19988 consensus_queue.cc:260] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.844521 19988 raft_consensus.cc:388] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:20.844592 19988 raft_consensus.cc:482] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:20.844652 19988 raft_consensus.cc:3037] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:20.847177 19988 raft_consensus.cc:504] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.847477 19988 leader_election.cc:302] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:20.847707 19988 leader_election.cc:288] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:20.847785 19990 raft_consensus.cc:2781] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:20.847925 19990 raft_consensus.cc:686] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:20.848184 19990 consensus_queue.cc:237] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:20.848279 19988 ts_tablet_manager.cc:1398] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.001s
I20240503 02:21:20.850183 19378 catalog_manager.cc:5484] T 0b8511a8bb1b4f2abe028021a8aac250 P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:21.382279 19378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36274:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:21:21.391423 19630 tablet_service.cc:1467] Processing CreateTablet for tablet 1b69c19cb4e74c5091aa47426e1f53b7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd1462f9f75c4606ab3672ee10581761]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.391490 19628 tablet_service.cc:1467] Processing CreateTablet for tablet 975a6cc379ce4dc5bf28411d4c8eb934 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd1462f9f75c4606ab3672ee10581761]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.391464 19629 tablet_service.cc:1467] Processing CreateTablet for tablet 60f56c936e8a4defb8b569b99f231569 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd1462f9f75c4606ab3672ee10581761]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.391916 19630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b69c19cb4e74c5091aa47426e1f53b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.392158 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975a6cc379ce4dc5bf28411d4c8eb934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.392377 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f56c936e8a4defb8b569b99f231569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.398511 20022 tablet_bootstrap.cc:492] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:21.400106 20022 tablet_bootstrap.cc:654] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:21.401973 20022 tablet_bootstrap.cc:492] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:21.402108 20022 ts_tablet_manager.cc:1367] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:21.402870 20022 raft_consensus.cc:348] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.403018 20022 raft_consensus.cc:374] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:21.403127 20022 raft_consensus.cc:729] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:21.403374 20022 consensus_queue.cc:260] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.403538 20022 raft_consensus.cc:388] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:21.403620 20022 raft_consensus.cc:482] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:21.403729 20022 raft_consensus.cc:3037] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:21.405716 20022 raft_consensus.cc:504] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.405956 20022 leader_election.cc:302] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:21.406342 20022 leader_election.cc:288] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:21.406522 20024 raft_consensus.cc:2781] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:21.406940 20022 ts_tablet_manager.cc:1398] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:21:21.407244 20022 tablet_bootstrap.cc:492] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:21.407217 20024 raft_consensus.cc:686] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:21.407511 20024 consensus_queue.cc:237] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.408756 20022 tablet_bootstrap.cc:654] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:21.410121 19378 catalog_manager.cc:5484] T 1b69c19cb4e74c5091aa47426e1f53b7 P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:21.411285 20022 tablet_bootstrap.cc:492] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:21.411463 20022 ts_tablet_manager.cc:1367] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:21.412271 20022 raft_consensus.cc:348] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.412437 20022 raft_consensus.cc:374] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:21.412525 20022 raft_consensus.cc:729] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:21.412806 20022 consensus_queue.cc:260] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.412989 20022 raft_consensus.cc:388] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:21.413103 20022 raft_consensus.cc:482] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:21.413211 20022 raft_consensus.cc:3037] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:21.415400 20022 raft_consensus.cc:504] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.415623 20022 leader_election.cc:302] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:21.415820 20022 leader_election.cc:288] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:21.415910 20026 raft_consensus.cc:2781] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:21.416100 20026 raft_consensus.cc:686] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:21.416250 20022 ts_tablet_manager.cc:1398] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:21:21.416523 20022 tablet_bootstrap.cc:492] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:21.416443 20026 consensus_queue.cc:237] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.418424 20022 tablet_bootstrap.cc:654] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:21.420388 20022 tablet_bootstrap.cc:492] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:21.420338 19378 catalog_manager.cc:5484] T 60f56c936e8a4defb8b569b99f231569 P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:21.420516 20022 ts_tablet_manager.cc:1367] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:21:21.421118 20022 raft_consensus.cc:348] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.421290 20022 raft_consensus.cc:374] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:21.421406 20022 raft_consensus.cc:729] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:21.421639 20022 consensus_queue.cc:260] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.421849 20022 raft_consensus.cc:388] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:21.421943 20022 raft_consensus.cc:482] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:21.422070 20022 raft_consensus.cc:3037] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:21.424224 20022 raft_consensus.cc:504] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.424469 20022 leader_election.cc:302] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:21.424635 20022 leader_election.cc:288] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:21.424711 20026 raft_consensus.cc:2781] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:21.424916 20026 raft_consensus.cc:686] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:21.425100 20022 ts_tablet_manager.cc:1398] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.002s
I20240503 02:21:21.425153 20026 consensus_queue.cc:237] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:21.427290 19378 catalog_manager.cc:5484] T 975a6cc379ce4dc5bf28411d4c8eb934 P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:21.971323 19378 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36274:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:21:21.980156 19499 tablet_service.cc:1467] Processing CreateTablet for tablet 67c7d30a560c4baebd755949ec0f9dc0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d33decd47d94f00ab6374a6837cc79f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.980298 19498 tablet_service.cc:1467] Processing CreateTablet for tablet 55084cebac0c4fb7a722dfd7e8bfd216 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d33decd47d94f00ab6374a6837cc79f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.980662 19499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c7d30a560c4baebd755949ec0f9dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.980944 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55084cebac0c4fb7a722dfd7e8bfd216. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.980176 19497 tablet_service.cc:1467] Processing CreateTablet for tablet a129f2eb24504b27b06ea8f0d035048e (DEFAULT_TABLE table=rebalance_test_table_2 [id=9d33decd47d94f00ab6374a6837cc79f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:21.981772 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a129f2eb24504b27b06ea8f0d035048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:21.987768 20057 tablet_bootstrap.cc:492] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:21.989128 20057 tablet_bootstrap.cc:654] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:21.990957 20057 tablet_bootstrap.cc:492] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:21.991101 20057 ts_tablet_manager.cc:1367] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:21.991699 20057 raft_consensus.cc:348] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:21.991850 20057 raft_consensus.cc:374] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:21.991948 20057 raft_consensus.cc:729] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:21.992189 20057 consensus_queue.cc:260] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:21.992352 20057 raft_consensus.cc:388] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:21.992431 20057 raft_consensus.cc:482] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:21.992537 20057 raft_consensus.cc:3037] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:21.994762 20057 raft_consensus.cc:504] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:21.995007 20057 leader_election.cc:302] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:21.995467 20057 leader_election.cc:288] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:21.995637 20059 raft_consensus.cc:2781] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:21.996151 20059 raft_consensus.cc:686] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:21.996476 20057 ts_tablet_manager.cc:1398] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.005s	user 0.006s	sys 0.000s
I20240503 02:21:21.996505 20059 consensus_queue.cc:237] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:21.996757 20057 tablet_bootstrap.cc:492] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:21.998618 20057 tablet_bootstrap.cc:654] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:21.998992 19378 catalog_manager.cc:5484] T 67c7d30a560c4baebd755949ec0f9dc0 P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:22.000488 20057 tablet_bootstrap.cc:492] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:22.000662 20057 ts_tablet_manager.cc:1367] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:22.001406 20057 raft_consensus.cc:348] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.001559 20057 raft_consensus.cc:374] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:22.001652 20057 raft_consensus.cc:729] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:22.001914 20057 consensus_queue.cc:260] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.002090 20057 raft_consensus.cc:388] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:22.002202 20057 raft_consensus.cc:482] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:22.002303 20057 raft_consensus.cc:3037] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:22.004580 20057 raft_consensus.cc:504] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.004876 20057 leader_election.cc:302] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:22.005112 20057 leader_election.cc:288] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:22.005177 20059 raft_consensus.cc:2781] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:22.005486 20059 raft_consensus.cc:686] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:22.005676 20057 ts_tablet_manager.cc:1398] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:21:22.005730 20059 consensus_queue.cc:237] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.005949 20057 tablet_bootstrap.cc:492] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67: Bootstrap starting.
I20240503 02:21:22.008325 20057 tablet_bootstrap.cc:654] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:22.008940 19378 catalog_manager.cc:5484] T a129f2eb24504b27b06ea8f0d035048e P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:22.010188 20057 tablet_bootstrap.cc:492] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67: No bootstrap required, opened a new log
I20240503 02:21:22.010358 20057 ts_tablet_manager.cc:1367] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:22.011000 20057 raft_consensus.cc:348] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.011178 20057 raft_consensus.cc:374] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:22.011282 20057 raft_consensus.cc:729] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Initialized, Role: FOLLOWER
I20240503 02:21:22.011516 20057 consensus_queue.cc:260] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.011708 20057 raft_consensus.cc:388] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:22.011793 20057 raft_consensus.cc:482] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:22.011902 20057 raft_consensus.cc:3037] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:22.014027 20057 raft_consensus.cc:504] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.014266 20057 leader_election.cc:302] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e4fb3769f34dfda47462f0550e8e67; no voters: 
I20240503 02:21:22.014461 20057 leader_election.cc:288] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:22.014541 20059 raft_consensus.cc:2781] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:22.014722 20059 raft_consensus.cc:686] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [term 1 LEADER]: Becoming Leader. State: Replica: e0e4fb3769f34dfda47462f0550e8e67, State: Running, Role: LEADER
I20240503 02:21:22.014935 20057 ts_tablet_manager.cc:1398] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:21:22.014982 20059 consensus_queue.cc:237] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } }
I20240503 02:21:22.017258 19378 catalog_manager.cc:5484] T 55084cebac0c4fb7a722dfd7e8bfd216 P e0e4fb3769f34dfda47462f0550e8e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e4fb3769f34dfda47462f0550e8e67 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "e0e4fb3769f34dfda47462f0550e8e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e4fb3769f34dfda47462f0550e8e67" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 38371 } } }
I20240503 02:21:22.541824 19377 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36274:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240503 02:21:22.550310 19629 tablet_service.cc:1467] Processing CreateTablet for tablet bcf113041575498a844fad1bf1eba6f6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f0f5eba27edf4993b685addcc93aefb6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:22.550522 19628 tablet_service.cc:1467] Processing CreateTablet for tablet bb2e37160fa04cae8c1d681b8b0395ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=f0f5eba27edf4993b685addcc93aefb6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:22.550825 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf113041575498a844fad1bf1eba6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:22.551133 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2e37160fa04cae8c1d681b8b0395ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:22.551542 19630 tablet_service.cc:1467] Processing CreateTablet for tablet ddfc7b9467f243a5a6346b26ae182482 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f0f5eba27edf4993b685addcc93aefb6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:22.551925 19630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddfc7b9467f243a5a6346b26ae182482. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:22.556437 20092 tablet_bootstrap.cc:492] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:22.558236 20092 tablet_bootstrap.cc:654] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:22.559883 20092 tablet_bootstrap.cc:492] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:22.560009 20092 ts_tablet_manager.cc:1367] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:22.560642 20092 raft_consensus.cc:348] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.560784 20092 raft_consensus.cc:374] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:22.560853 20092 raft_consensus.cc:729] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:22.561079 20092 consensus_queue.cc:260] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.561233 20092 raft_consensus.cc:388] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:22.561312 20092 raft_consensus.cc:482] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:22.561473 20092 raft_consensus.cc:3037] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:22.563323 20092 raft_consensus.cc:504] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.563546 20092 leader_election.cc:302] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:22.563946 20092 leader_election.cc:288] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:22.564143 20094 raft_consensus.cc:2781] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:22.564697 20092 ts_tablet_manager.cc:1398] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20240503 02:21:22.564870 20094 raft_consensus.cc:686] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:22.564931 20092 tablet_bootstrap.cc:492] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:22.565207 20094 consensus_queue.cc:237] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.566658 20092 tablet_bootstrap.cc:654] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:22.567910 19377 catalog_manager.cc:5484] T bcf113041575498a844fad1bf1eba6f6 P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:22.570648 20092 tablet_bootstrap.cc:492] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:22.570845 20092 ts_tablet_manager.cc:1367] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240503 02:21:22.571586 20092 raft_consensus.cc:348] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.571782 20092 raft_consensus.cc:374] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:22.571868 20092 raft_consensus.cc:729] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:22.572098 20092 consensus_queue.cc:260] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.572244 20092 raft_consensus.cc:388] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:22.572347 20092 raft_consensus.cc:482] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:22.572438 20092 raft_consensus.cc:3037] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:22.574402 20092 raft_consensus.cc:504] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.574640 20092 leader_election.cc:302] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:22.574831 20092 leader_election.cc:288] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:22.574918 20094 raft_consensus.cc:2781] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:22.575161 20094 raft_consensus.cc:686] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:22.575335 20092 ts_tablet_manager.cc:1398] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.004s
I20240503 02:21:22.575402 20094 consensus_queue.cc:237] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.575595 20092 tablet_bootstrap.cc:492] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Bootstrap starting.
I20240503 02:21:22.577483 20092 tablet_bootstrap.cc:654] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:22.577689 19377 catalog_manager.cc:5484] T bb2e37160fa04cae8c1d681b8b0395ce P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:22.579191 20092 tablet_bootstrap.cc:492] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50: No bootstrap required, opened a new log
I20240503 02:21:22.579334 20092 ts_tablet_manager.cc:1367] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:21:22.579979 20092 raft_consensus.cc:348] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.580125 20092 raft_consensus.cc:374] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:22.580200 20092 raft_consensus.cc:729] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Initialized, Role: FOLLOWER
I20240503 02:21:22.580452 20092 consensus_queue.cc:260] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.580663 20092 raft_consensus.cc:388] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:22.580761 20092 raft_consensus.cc:482] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:22.580879 20092 raft_consensus.cc:3037] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:22.583343 20092 raft_consensus.cc:504] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.583580 20092 leader_election.cc:302] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1ee1626ca1d4a52a2dda4ba42b8ce50; no voters: 
I20240503 02:21:22.583760 20092 leader_election.cc:288] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:22.583864 20094 raft_consensus.cc:2781] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:22.584057 20094 raft_consensus.cc:686] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [term 1 LEADER]: Becoming Leader. State: Replica: b1ee1626ca1d4a52a2dda4ba42b8ce50, State: Running, Role: LEADER
I20240503 02:21:22.584256 20092 ts_tablet_manager.cc:1398] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:21:22.584317 20094 consensus_queue.cc:237] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } }
I20240503 02:21:22.586288 19378 catalog_manager.cc:5484] T ddfc7b9467f243a5a6346b26ae182482 P b1ee1626ca1d4a52a2dda4ba42b8ce50 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ee1626ca1d4a52a2dda4ba42b8ce50 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1ee1626ca1d4a52a2dda4ba42b8ce50" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 40887 } } }
I20240503 02:21:23.115679 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:35137
--local_ip_for_outbound_sockets=127.20.218.67
--tserver_master_addrs=127.20.218.126:46685
--webserver_port=39191
--webserver_interface=127.20.218.67
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:21:23.473768 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:23.474112 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:23.474259 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:23.481978 20124 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:23.482167 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:23.482343 20124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:21:23.491823 20124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:35137
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=39191
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:23.493271 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:23.495126 20124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:23.507419 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:23.507452 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:23.507506 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:23.508134 20124 server_base.cc:1034] running on GCE node
I20240503 02:21:23.508801 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:23.509871 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:23.511080 20124 hybrid_clock.cc:648] HybridClock initialized: now 1714702883511024 us; error 43 us; skew 500 ppm
I20240503 02:21:23.511428 20124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:23.513928 20124 webserver.cc:460] Webserver started at http://127.20.218.67:39191/ using document root <none> and password file <none>
I20240503 02:21:23.514704 20124 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:23.514858 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:23.520634 20124 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240503 02:21:23.523639 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:23.525103 20124 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:23.525283 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "67ed198a70f54a919d2be9d42980ef7b"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:23.526198 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:23.545987 20124 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:23.546854 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:23.547155 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:23.548197 20124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:23.549952 20124 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:23.550056 20124 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:23.550199 20124 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:23.550254 20124 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:23.583598 20124 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:35137
I20240503 02:21:23.583640 20250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:35137 every 8 connection(s)
I20240503 02:21:23.585122 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:21:23.591106 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20124
I20240503 02:21:23.591884 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:34269
--local_ip_for_outbound_sockets=127.20.218.68
--tserver_master_addrs=127.20.218.126:46685
--webserver_port=39503
--webserver_interface=127.20.218.68
--builtin_ntp_servers=127.20.218.84:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:21:23.602380 20251 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:23.602737 20251 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:23.603514 20251 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:23.605145 19377 ts_manager.cc:194] Re-registered known tserver with Master: 67ed198a70f54a919d2be9d42980ef7b (127.20.218.67:35137)
I20240503 02:21:23.606011 19377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:57933
W20240503 02:21:23.951805 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:23.952144 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:23.952286 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:23.960038 20255 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:21:23.960199 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:23.960359 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:23.969771 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:34269
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=39503
--tserver_master_addrs=127.20.218.126:46685
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:23.971144 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:23.972913 20255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:23.985080 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:23.985128 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:23.985157 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:23.985978 20255 server_base.cc:1034] running on GCE node
I20240503 02:21:23.986585 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:23.987563 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:23.988765 20255 hybrid_clock.cc:648] HybridClock initialized: now 1714702883988712 us; error 54 us; skew 500 ppm
I20240503 02:21:23.989073 20255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:23.991549 20255 webserver.cc:460] Webserver started at http://127.20.218.68:39503/ using document root <none> and password file <none>
I20240503 02:21:23.992291 20255 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:23.992434 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:23.997900 20255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:21:24.000694 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:24.002137 20255 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:24.002300 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "95827103554543fd829afd3e8644de1d"
format_stamp: "Formatted at 2024-05-03 02:21:16 on dist-test-slave-81lq"
I20240503 02:21:24.003126 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:24.024555 20255 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:24.025696 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:24.026000 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:24.027057 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:24.028662 20255 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:24.028738 20255 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:24.028823 20255 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:24.028870 20255 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:24.062638 20255 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:34269
I20240503 02:21:24.062691 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:34269 every 8 connection(s)
I20240503 02:21:24.064126 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:24.071120 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20255
I20240503 02:21:24.088747 20382 heartbeater.cc:347] Connected to a master server at 127.20.218.126:46685
I20240503 02:21:24.089123 20382 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:24.090255 20382 heartbeater.cc:511] Master 127.20.218.126:46685 requested a full tablet report, sending...
I20240503 02:21:24.091969 19377 ts_manager.cc:194] Re-registered known tserver with Master: 95827103554543fd829afd3e8644de1d (127.20.218.68:34269)
I20240503 02:21:24.093087 19377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:56739
I20240503 02:21:24.576623 19498 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:24.580045 19630 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:24.581532 20186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.584640 20317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.609144 20251 heartbeater.cc:503] Master 127.20.218.126:46685 was elected leader, sending a full tablet report...
I20240503 02:21:24.789590 20317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.790772 20186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.806157 19630 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:24.806151 19498 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:24.960974 20317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.963449 20186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:24.975036 19630 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:24.975783 19498 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 02:21:25.095747 20382 heartbeater.cc:503] Master 127.20.218.126:46685 was elected leader, sending a full tablet report...
I20240503 02:21:25.232642 20186 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:25.240064 20317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:25.242400 19630 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:21:25.259054 19498 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 5720469d90d84bbe82d6b2d18f5589d4 | 127.20.218.126:46685 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                                |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                               | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                  | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                               | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                 | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                                | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value        |         Master
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:40869 | all 1 server(s) checked
 time_source         | builtin             | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
 67ed198a70f54a919d2be9d42980ef7b | 127.20.218.67:35137 | HEALTHY | <none>   |       0        |       0
 95827103554543fd829afd3e8644de1d | 127.20.218.68:34269 | HEALTHY | <none>   |       0        |       0
 b1ee1626ca1d4a52a2dda4ba42b8ce50 | 127.20.218.66:40887 | HEALTHY | <none>   |       7        |       0
 e0e4fb3769f34dfda47462f0550e8e67 | 127.20.218.65:38371 | HEALTHY | <none>   |       6        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                              |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                            | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.20.218.65                                                                                                                                                                                  | experimental          | 127.20.218.65:38371
 local_ip_for_outbound_sockets            | 127.20.218.66                                                                                                                                                                                  | experimental          | 127.20.218.66:40887
 local_ip_for_outbound_sockets            | 127.20.218.67                                                                                                                                                                                  | experimental          | 127.20.218.67:35137
 local_ip_for_outbound_sockets            | 127.20.218.68                                                                                                                                                                                  | experimental          | 127.20.218.68:34269
 never_fsync                              | true                                                                                                                                                                                           | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                              | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                           | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                             | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.20.218.65:38371
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.20.218.66:40887
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.20.218.67:35137
 server_dump_info_path                    | /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.20.218.68:34269

Flags of checked categories for Tablet Server:
        Flag         |        Value        |      Tablet Server
---------------------+---------------------+-------------------------
 builtin_ntp_servers | 127.20.218.84:40869 | all 4 server(s) checked
 time_source         | builtin             | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 6
 Third Quartile | 7
 Maximum        | 7

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240503 02:21:25.315002 21353 log_verifier.cc:126] Checking tablet 0b8511a8bb1b4f2abe028021a8aac250
I20240503 02:21:25.323688 21353 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 0b8511a8bb1b4f2abe028021a8aac250
I20240503 02:21:25.323799 21353 log_verifier.cc:126] Checking tablet 1b69c19cb4e74c5091aa47426e1f53b7
I20240503 02:21:25.331954 21353 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 1b69c19cb4e74c5091aa47426e1f53b7
I20240503 02:21:25.332054 21353 log_verifier.cc:126] Checking tablet 55084cebac0c4fb7a722dfd7e8bfd216
I20240503 02:21:25.340027 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 55084cebac0c4fb7a722dfd7e8bfd216
I20240503 02:21:25.340214 21353 log_verifier.cc:126] Checking tablet 5bd107a6b54646e6b93e03959c7a1637
I20240503 02:21:25.348176 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5bd107a6b54646e6b93e03959c7a1637
I20240503 02:21:25.348284 21353 log_verifier.cc:126] Checking tablet 60f56c936e8a4defb8b569b99f231569
I20240503 02:21:25.356119 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 60f56c936e8a4defb8b569b99f231569
I20240503 02:21:25.356220 21353 log_verifier.cc:126] Checking tablet 67c7d30a560c4baebd755949ec0f9dc0
I20240503 02:21:25.364200 21353 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 67c7d30a560c4baebd755949ec0f9dc0
I20240503 02:21:25.364308 21353 log_verifier.cc:126] Checking tablet 975a6cc379ce4dc5bf28411d4c8eb934
I20240503 02:21:25.372056 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 975a6cc379ce4dc5bf28411d4c8eb934
I20240503 02:21:25.372161 21353 log_verifier.cc:126] Checking tablet 9dac7d8b040d45fbbda6596176a40e3d
I20240503 02:21:25.380102 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9dac7d8b040d45fbbda6596176a40e3d
I20240503 02:21:25.380208 21353 log_verifier.cc:126] Checking tablet a129f2eb24504b27b06ea8f0d035048e
I20240503 02:21:25.388034 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a129f2eb24504b27b06ea8f0d035048e
I20240503 02:21:25.388139 21353 log_verifier.cc:126] Checking tablet bb2e37160fa04cae8c1d681b8b0395ce
I20240503 02:21:25.395932 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet bb2e37160fa04cae8c1d681b8b0395ce
I20240503 02:21:25.396044 21353 log_verifier.cc:126] Checking tablet bcf113041575498a844fad1bf1eba6f6
I20240503 02:21:25.404451 21353 log_verifier.cc:177] Verified matching terms for 2 ops in tablet bcf113041575498a844fad1bf1eba6f6
I20240503 02:21:25.404551 21353 log_verifier.cc:126] Checking tablet ddfc7b9467f243a5a6346b26ae182482
I20240503 02:21:25.412485 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ddfc7b9467f243a5a6346b26ae182482
I20240503 02:21:25.412598 21353 log_verifier.cc:126] Checking tablet ea65793ff1dc42fdb30fdf6bf2b12252
I20240503 02:21:25.420507 21353 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ea65793ff1dc42fdb30fdf6bf2b12252
I20240503 02:21:25.421530 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 19437
I20240503 02:21:25.422971 19559 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:25.523881 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 19437
I20240503 02:21:25.545096 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 19568
I20240503 02:21:25.546305 19690 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:25.648795 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 19568
W20240503 02:21:25.670918 19363 connection.cc:535] server connection from 127.20.218.66:34267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:25.671110 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20124
I20240503 02:21:25.677870 20246 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:25.766357 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20124
I20240503 02:21:25.785748 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20255
I20240503 02:21:25.792049 20377 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:25.881805 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20255
I20240503 02:21:25.900977 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 19347
I20240503 02:21:25.902197 19408 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:26.022220 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 19347
2024-05-03T02:21:26Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (12109 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12109 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240503 02:21:26.081909 21353 test_util.cc:268] Using random seed: 1617663323
I20240503 02:21:26.084017 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:21:26.084095 21353 ts_itest-base.cc:113] --------------
I20240503 02:21:26.084126 21353 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:21:26.084169 21353 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:21:26.084242 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:21:26Z Disabled control of system clock
I20240503 02:21:26.142449 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:38245
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:38245
--location_mapping_cmd=/tmp/dist-test-taskEtZc7o/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:21:26.499922 20534 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:21:26.500303 20534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:26.500416 20534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:26.500517 20534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:26.508203 20534 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:26.508342 20534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:21:26.508455 20534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:26.508514 20534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:21:26.508610 20534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:21:26.518046 20534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskEtZc7o/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:38245
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:26.519421 20534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:26.521204 20534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:26.532406 20542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:26.532367 20540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:26.532349 20539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:26.533106 20534 server_base.cc:1034] running on GCE node
I20240503 02:21:26.533946 20534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:26.535427 20534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:26.536666 20534 hybrid_clock.cc:648] HybridClock initialized: now 1714702886536607 us; error 43 us; skew 500 ppm
I20240503 02:21:26.537031 20534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:26.539422 20534 webserver.cc:460] Webserver started at http://127.20.218.126:42929/ using document root <none> and password file <none>
I20240503 02:21:26.540180 20534 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:26.540316 20534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:26.540620 20534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:26.543411 20534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fa895429be44453bb61a2ff652eb633d"
format_stamp: "Formatted at 2024-05-03 02:21:26 on dist-test-slave-81lq"
I20240503 02:21:26.544239 20534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fa895429be44453bb61a2ff652eb633d"
format_stamp: "Formatted at 2024-05-03 02:21:26 on dist-test-slave-81lq"
I20240503 02:21:26.549770 20534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20240503 02:21:26.553380 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:26.554751 20534 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:21:26.554935 20534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fa895429be44453bb61a2ff652eb633d"
format_stamp: "Formatted at 2024-05-03 02:21:26 on dist-test-slave-81lq"
I20240503 02:21:26.555148 20534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:26.574357 20534 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:26.575271 20534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:26.575567 20534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:26.594717 20534 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:38245
I20240503 02:21:26.594739 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:38245 every 8 connection(s)
I20240503 02:21:26.596230 20534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:21:26.601577 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:26.616511 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d: Bootstrap starting.
I20240503 02:21:26.621671 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:26.623176 20600 log.cc:824] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:26.626751 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d: No bootstrap required, opened a new log
I20240503 02:21:26.637084 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } }
I20240503 02:21:26.637547 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:26.637666 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa895429be44453bb61a2ff652eb633d, State: Initialized, Role: FOLLOWER
I20240503 02:21:26.638334 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } }
I20240503 02:21:26.638550 20600 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:26.638689 20600 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:26.638831 20600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:26.640977 20600 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } }
I20240503 02:21:26.641541 20600 leader_election.cc:302] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa895429be44453bb61a2ff652eb633d; no voters: 
I20240503 02:21:26.642269 20600 leader_election.cc:288] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:26.642400 20603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:26.643759 20603 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [term 1 LEADER]: Becoming Leader. State: Replica: fa895429be44453bb61a2ff652eb633d, State: Running, Role: LEADER
I20240503 02:21:26.644429 20603 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } }
I20240503 02:21:26.644862 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:21:26.647608 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa895429be44453bb61a2ff652eb633d. Latest consensus state: current_term: 1 leader_uuid: "fa895429be44453bb61a2ff652eb633d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } } }
I20240503 02:21:26.648022 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:26.648536 20604 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa895429be44453bb61a2ff652eb633d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa895429be44453bb61a2ff652eb633d" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 38245 } } }
I20240503 02:21:26.648846 20604 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:26.651160 20610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:21:26.651362 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20534
I20240503 02:21:26.651557 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:21:26.670866 20610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:21:26.690166 20610 catalog_manager.cc:1347] Generated new cluster ID: 3e328b8e6dce4c749a89272c3232d28e
I20240503 02:21:26.690459 20610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:21:26.707787 20610 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:21:26.709143 20610 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:21:26.732815 20610 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fa895429be44453bb61a2ff652eb633d: Generated new TSK 0
I20240503 02:21:26.733808 20610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:21:26.749508 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:21:27.107815 20624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:27.108161 20624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:27.108255 20624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:27.115955 20624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:27.116113 20624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:27.116241 20624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:21:27.125706 20624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:27.127069 20624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:27.128798 20624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:27.141176 20630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:27.141074 20629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:27.141403 20632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:27.142383 20624 server_base.cc:1034] running on GCE node
I20240503 02:21:27.143052 20624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:27.144129 20624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:27.145458 20624 hybrid_clock.cc:648] HybridClock initialized: now 1714702887145390 us; error 63 us; skew 500 ppm
I20240503 02:21:27.145817 20624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:27.152328 20624 webserver.cc:460] Webserver started at http://127.20.218.65:39367/ using document root <none> and password file <none>
I20240503 02:21:27.153188 20624 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:27.153319 20624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:27.153681 20624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:27.156435 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5ba5197bbdb144878029533fd45d53b3"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.157315 20624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5ba5197bbdb144878029533fd45d53b3"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.162904 20624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:21:27.166451 20638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.167790 20624 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240503 02:21:27.167961 20624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5ba5197bbdb144878029533fd45d53b3"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.168179 20624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:27.194043 20624 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:27.194984 20624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:27.195278 20624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:27.196312 20624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:27.197978 20624 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:27.198077 20624 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.198211 20624 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:27.198297 20624 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.229445 20624 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:43579
I20240503 02:21:27.229502 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:43579 every 8 connection(s)
I20240503 02:21:27.230854 20624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:21:27.234316 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20624
I20240503 02:21:27.234563 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:21:27.238198 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:27.245011 20751 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:27.245383 20751 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:27.246381 20751 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:27.302093 20565 ts_manager.cc:194] Registered new tserver with Master: 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:27.303917 20565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:35583
W20240503 02:21:27.596050 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:27.596362 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:27.596488 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:27.604439 20755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:27.604576 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:27.604743 20755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:21:27.614207 20755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:27.615585 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:27.617362 20755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:27.629089 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:27.629221 20762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:27.629118 20764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:27.629848 20755 server_base.cc:1034] running on GCE node
I20240503 02:21:27.630483 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:27.631448 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:27.632653 20755 hybrid_clock.cc:648] HybridClock initialized: now 1714702887632608 us; error 37 us; skew 500 ppm
I20240503 02:21:27.633000 20755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:27.635313 20755 webserver.cc:460] Webserver started at http://127.20.218.66:35163/ using document root <none> and password file <none>
I20240503 02:21:27.636080 20755 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:27.636205 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:27.636518 20755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:27.639221 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7836d9d893434d7a99d61974b10dd946"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.640030 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7836d9d893434d7a99d61974b10dd946"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.645422 20755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240503 02:21:27.648897 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.650352 20755 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240503 02:21:27.650530 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7836d9d893434d7a99d61974b10dd946"
format_stamp: "Formatted at 2024-05-03 02:21:27 on dist-test-slave-81lq"
I20240503 02:21:27.650751 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:27.667050 20755 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:27.667932 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:27.668220 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:27.669191 20755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:27.670893 20755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:27.671007 20755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.671128 20755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:27.671201 20755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:27.704864 20755 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:33349
I20240503 02:21:27.704927 20882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:33349 every 8 connection(s)
I20240503 02:21:27.706283 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:21:27.716365 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20755
I20240503 02:21:27.716567 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:21:27.719739 20883 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:27.720106 20883 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:27.720340 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:27.721035 20883 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:27.768059 20564 ts_manager.cc:194] Registered new tserver with Master: 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:27.769454 20564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:44711
W20240503 02:21:28.072202 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:28.072585 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:28.072737 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:28.080521 20887 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:28.080693 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:28.080864 20887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:21:28.090420 20887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:28.091784 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:28.093628 20887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:28.105363 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:28.106040 20893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:28.106163 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:28.106499 20887 server_base.cc:1034] running on GCE node
I20240503 02:21:28.107146 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:28.108150 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:28.109474 20887 hybrid_clock.cc:648] HybridClock initialized: now 1714702888109413 us; error 54 us; skew 500 ppm
I20240503 02:21:28.109818 20887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:28.112305 20887 webserver.cc:460] Webserver started at http://127.20.218.67:38515/ using document root <none> and password file <none>
I20240503 02:21:28.113108 20887 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:28.113246 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:28.113606 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:28.116341 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c184c96b994d4707919e20333d152788"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.117218 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c184c96b994d4707919e20333d152788"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.122772 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:21:28.126410 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.127858 20887 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:21:28.128062 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c184c96b994d4707919e20333d152788"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.128285 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:28.142468 20887 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:28.143342 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:28.143610 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:28.144569 20887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:28.146235 20887 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:28.146345 20887 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.146467 20887 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:28.146545 20887 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.180610 20887 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:41081
I20240503 02:21:28.180670 21014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:41081 every 8 connection(s)
I20240503 02:21:28.182068 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:21:28.184505 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 20887
I20240503 02:21:28.184700 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:21:28.188118 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:28.195853 21015 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:28.196203 21015 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:28.197101 21015 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:28.247475 20564 ts_manager.cc:194] Registered new tserver with Master: c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:28.248427 20564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:52405
I20240503 02:21:28.307240 20751 heartbeater.cc:503] Master 127.20.218.126:38245 was elected leader, sending a full tablet report...
W20240503 02:21:28.542441 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:28.542779 21018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:28.542946 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:28.550873 21018 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:28.551017 21018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:28.551174 21018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:28.560705 21018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:28.562187 21018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:28.564086 21018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:28.577077 21024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:28.577184 21025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:28.577124 21027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:28.582167 21018 server_base.cc:1034] running on GCE node
I20240503 02:21:28.582862 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:28.583887 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:28.585119 21018 hybrid_clock.cc:648] HybridClock initialized: now 1714702888585007 us; error 105 us; skew 500 ppm
I20240503 02:21:28.585539 21018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:28.587906 21018 webserver.cc:460] Webserver started at http://127.20.218.68:39727/ using document root <none> and password file <none>
I20240503 02:21:28.588639 21018 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:28.588747 21018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:28.589025 21018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:28.591745 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "16c046bf08ee4bd684ff0fcb5b158d4d"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.592581 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "16c046bf08ee4bd684ff0fcb5b158d4d"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.599241 21018 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20240503 02:21:28.603850 21033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.605433 21018 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:28.605597 21018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "16c046bf08ee4bd684ff0fcb5b158d4d"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:28.605793 21018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:28.629022 21018 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:28.629928 21018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:28.630193 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:28.631166 21018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:28.632731 21018 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:28.632822 21018 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.632912 21018 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:28.632989 21018 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:28.676122 21018 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:36887
I20240503 02:21:28.676177 21145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:36887 every 8 connection(s)
I20240503 02:21:28.677515 21018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:28.684592 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21018
I20240503 02:21:28.684763 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:21:28.687995 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:0
--local_ip_for_outbound_sockets=127.20.218.69
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:28.691484 21146 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:28.691826 21146 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:28.692880 21146 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:28.741852 20564 ts_manager.cc:194] Registered new tserver with Master: 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887)
I20240503 02:21:28.743216 20564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:33565
I20240503 02:21:28.772300 20883 heartbeater.cc:503] Master 127.20.218.126:38245 was elected leader, sending a full tablet report...
W20240503 02:21:29.052848 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:29.053205 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:29.053398 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:29.061251 21150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:29.061450 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:29.061620 21150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:21:29.071472 21150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=0
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:29.072994 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:29.074868 21150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:29.087066 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:29.087126 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:29.087359 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:29.087957 21150 server_base.cc:1034] running on GCE node
I20240503 02:21:29.088670 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:29.089702 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:29.090916 21150 hybrid_clock.cc:648] HybridClock initialized: now 1714702889090861 us; error 50 us; skew 500 ppm
I20240503 02:21:29.091274 21150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:29.093772 21150 webserver.cc:460] Webserver started at http://127.20.218.69:37781/ using document root <none> and password file <none>
I20240503 02:21:29.094548 21150 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:29.094661 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:29.094996 21150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:29.097803 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "46b0cc47f0274f63a284f07b49cc6da5"
format_stamp: "Formatted at 2024-05-03 02:21:29 on dist-test-slave-81lq"
I20240503 02:21:29.098661 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "46b0cc47f0274f63a284f07b49cc6da5"
format_stamp: "Formatted at 2024-05-03 02:21:29 on dist-test-slave-81lq"
I20240503 02:21:29.104099 21150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:21:29.107682 21164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:29.109246 21150 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:29.109467 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "46b0cc47f0274f63a284f07b49cc6da5"
format_stamp: "Formatted at 2024-05-03 02:21:29 on dist-test-slave-81lq"
I20240503 02:21:29.109686 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:29.143769 21150 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:29.144662 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:29.145000 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:29.146059 21150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:29.147608 21150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:29.147709 21150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:29.147832 21150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:29.147907 21150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:29.179730 21150 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:44045
I20240503 02:21:29.179808 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:44045 every 8 connection(s)
I20240503 02:21:29.181483 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:21:29.186159 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21150
I20240503 02:21:29.186398 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:21:29.195832 21277 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:29.196154 21277 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:29.197007 21277 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:29.245836 20564 ts_manager.cc:194] Registered new tserver with Master: 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045)
I20240503 02:21:29.246871 20564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:53923
I20240503 02:21:29.251122 21015 heartbeater.cc:503] Master 127.20.218.126:38245 was elected leader, sending a full tablet report...
I20240503 02:21:29.256911 21353 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:21:29.274273 21353 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:21:29.274435 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21150
I20240503 02:21:29.283223 21272 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:29.364460 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21150
I20240503 02:21:29.381767 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21018
I20240503 02:21:29.390168 21141 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:29.471730 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21018
I20240503 02:21:33.241899 20564 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57184:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:33.249003 20564 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:33.287637 20816 tablet_service.cc:1467] Processing CreateTablet for tablet 080b798481e34ca69b0df5141e439567 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.288304 20817 tablet_service.cc:1467] Processing CreateTablet for tablet a69bdcc01c674a81a3cfd83e9eaaea94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.289518 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69bdcc01c674a81a3cfd83e9eaaea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.290762 20816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080b798481e34ca69b0df5141e439567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.292660 20686 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2e64b53d804601b7311747a3f8dce4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.294328 20686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2e64b53d804601b7311747a3f8dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.292716 20684 tablet_service.cc:1467] Processing CreateTablet for tablet 080b798481e34ca69b0df5141e439567 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.296100 20685 tablet_service.cc:1467] Processing CreateTablet for tablet a69bdcc01c674a81a3cfd83e9eaaea94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.296569 20818 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2e64b53d804601b7311747a3f8dce4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.297246 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2e64b53d804601b7311747a3f8dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.296972 20949 tablet_service.cc:1467] Processing CreateTablet for tablet 9c2e64b53d804601b7311747a3f8dce4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.297909 20948 tablet_service.cc:1467] Processing CreateTablet for tablet a69bdcc01c674a81a3cfd83e9eaaea94 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.298369 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 080b798481e34ca69b0df5141e439567 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:33.298604 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2e64b53d804601b7311747a3f8dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.299400 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080b798481e34ca69b0df5141e439567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.300861 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69bdcc01c674a81a3cfd83e9eaaea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.295809 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080b798481e34ca69b0df5141e439567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.305611 20685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69bdcc01c674a81a3cfd83e9eaaea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:33.310482 21301 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:33.316994 21301 tablet_bootstrap.cc:654] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.318974 21301 log.cc:824] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:33.326642 21301 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:33.329936 21301 ts_tablet_manager.cc:1367] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.020s	user 0.008s	sys 0.003s
I20240503 02:21:33.334100 21302 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:33.337096 21302 tablet_bootstrap.cc:654] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.338168 21304 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:33.339870 21302 log.cc:824] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:33.341885 21304 tablet_bootstrap.cc:654] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.342226 21301 raft_consensus.cc:348] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.342588 21301 raft_consensus.cc:374] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.342741 21301 raft_consensus.cc:729] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.343232 21302 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:33.343420 21301 consensus_queue.cc:260] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.343544 21302 ts_tablet_manager.cc:1367] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.000s
I20240503 02:21:33.343587 21304 log.cc:824] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:33.345067 21301 ts_tablet_manager.cc:1398] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.000s
I20240503 02:21:33.345715 21301 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:33.347545 21301 tablet_bootstrap.cc:654] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.356701 21302 raft_consensus.cc:348] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.357172 21302 raft_consensus.cc:374] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.357295 21302 raft_consensus.cc:729] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.358098 21301 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:33.358042 21302 consensus_queue.cc:260] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.358330 21301 ts_tablet_manager.cc:1367] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240503 02:21:33.359530 21301 raft_consensus.cc:348] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.359915 21301 raft_consensus.cc:374] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.360061 21301 raft_consensus.cc:729] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.360426 21302 ts_tablet_manager.cc:1398] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.007s
I20240503 02:21:33.360381 21301 consensus_queue.cc:260] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.361408 21302 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:33.361820 21304 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:33.361887 21301 ts_tablet_manager.cc:1398] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:33.362193 21304 ts_tablet_manager.cc:1367] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.012s
I20240503 02:21:33.362268 21301 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:33.364063 21301 tablet_bootstrap.cc:654] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.366142 21302 tablet_bootstrap.cc:654] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.368294 21302 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:33.368328 21301 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:33.368455 21302 ts_tablet_manager.cc:1367] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240503 02:21:33.368492 21301 ts_tablet_manager.cc:1367] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240503 02:21:33.369230 21301 raft_consensus.cc:348] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.369518 21301 raft_consensus.cc:374] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.369590 21301 raft_consensus.cc:729] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.369577 21302 raft_consensus.cc:348] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.369784 21302 raft_consensus.cc:374] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.369876 21302 raft_consensus.cc:729] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.369856 21301 consensus_queue.cc:260] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.370158 21302 consensus_queue.cc:260] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.370507 21301 ts_tablet_manager.cc:1398] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:21:33.370798 21302 ts_tablet_manager.cc:1398] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:33.371129 21302 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:33.371733 21304 raft_consensus.cc:348] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.372406 21304 raft_consensus.cc:374] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.372581 21304 raft_consensus.cc:729] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.372876 21302 tablet_bootstrap.cc:654] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.373627 21304 consensus_queue.cc:260] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.375931 21302 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:33.376101 21302 ts_tablet_manager.cc:1367] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:21:33.376917 21304 ts_tablet_manager.cc:1398] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.015s	user 0.013s	sys 0.000s
I20240503 02:21:33.376904 21302 raft_consensus.cc:348] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.377161 21302 raft_consensus.cc:374] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.377271 21302 raft_consensus.cc:729] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.377409 21304 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:33.377558 21302 consensus_queue.cc:260] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.378175 21302 ts_tablet_manager.cc:1398] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:33.379196 21304 tablet_bootstrap.cc:654] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.381393 21304 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:33.381634 21304 ts_tablet_manager.cc:1367] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240503 02:21:33.382536 21304 raft_consensus.cc:348] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.382722 21304 raft_consensus.cc:374] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.382812 21304 raft_consensus.cc:729] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.383028 21304 consensus_queue.cc:260] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.383531 21304 ts_tablet_manager.cc:1398] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:33.383831 21304 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:33.384585 21309 raft_consensus.cc:482] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:33.384836 21309 raft_consensus.cc:504] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.385556 21304 tablet_bootstrap.cc:654] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:33.387054 21309 leader_election.cc:288] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:33.387706 21304 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:33.387887 21304 ts_tablet_manager.cc:1367] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:21:33.388638 21304 raft_consensus.cc:348] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.388885 21304 raft_consensus.cc:374] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:33.388983 21304 raft_consensus.cc:729] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:33.389228 21304 consensus_queue.cc:260] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.389853 21304 ts_tablet_manager.cc:1398] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:33.400208 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69bdcc01c674a81a3cfd83e9eaaea94" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:33.400208 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69bdcc01c674a81a3cfd83e9eaaea94" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946" is_pre_election: true
I20240503 02:21:33.400723 20706 raft_consensus.cc:2454] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 0.
I20240503 02:21:33.400728 20838 raft_consensus.cc:2454] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 0.
I20240503 02:21:33.401391 20903 leader_election.cc:302] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:33.401798 21309 raft_consensus.cc:2781] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:33.401945 21309 raft_consensus.cc:482] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:33.402046 21309 raft_consensus.cc:3037] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.404155 21309 raft_consensus.cc:504] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.404842 21309 leader_election.cc:288] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:33.405445 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69bdcc01c674a81a3cfd83e9eaaea94" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:33.405489 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69bdcc01c674a81a3cfd83e9eaaea94" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:33.405685 20838 raft_consensus.cc:3037] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.405685 20706 raft_consensus.cc:3037] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.408905 20706 raft_consensus.cc:2454] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 1.
I20240503 02:21:33.408905 20838 raft_consensus.cc:2454] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 1.
I20240503 02:21:33.409430 20903 leader_election.cc:302] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:33.409797 21309 raft_consensus.cc:2781] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:33.410259 21309 raft_consensus.cc:686] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Becoming Leader. State: Replica: c184c96b994d4707919e20333d152788, State: Running, Role: LEADER
I20240503 02:21:33.410811 21309 consensus_queue.cc:237] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.415555 20564 catalog_manager.cc:5484] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 reported cstate change: term changed from 0 to 1, leader changed from <none> to c184c96b994d4707919e20333d152788 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } }
W20240503 02:21:33.437613 21016 tablet.cc:2369] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:21:33.462457 20884 tablet.cc:2369] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:21:33.487928 20752 tablet.cc:2369] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:21:33.618456 21306 raft_consensus.cc:482] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:33.618685 21306 raft_consensus.cc:504] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.620321 21306 leader_election.cc:288] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:33.627470 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2e64b53d804601b7311747a3f8dce4" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:33.627780 20706 raft_consensus.cc:2454] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:33.628023 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2e64b53d804601b7311747a3f8dce4" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:33.628451 20969 raft_consensus.cc:2454] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:33.628422 20774 leader_election.cc:302] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:33.628860 21306 raft_consensus.cc:2781] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:33.628978 21306 raft_consensus.cc:482] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:33.629082 21306 raft_consensus.cc:3037] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.631157 21306 raft_consensus.cc:504] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.631753 21306 leader_election.cc:288] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:33.632297 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2e64b53d804601b7311747a3f8dce4" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:33.632375 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c2e64b53d804601b7311747a3f8dce4" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:33.632560 20969 raft_consensus.cc:3037] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.632582 20706 raft_consensus.cc:3037] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.635500 20706 raft_consensus.cc:2454] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:33.635579 20969 raft_consensus.cc:2454] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:33.636009 20774 leader_election.cc:302] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:33.636332 21306 raft_consensus.cc:2781] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:33.636780 21306 raft_consensus.cc:686] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Leader. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:33.637265 21306 consensus_queue.cc:237] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.642020 20564 catalog_manager.cc:5484] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7836d9d893434d7a99d61974b10dd946 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:33.684541 21313 raft_consensus.cc:482] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:33.684774 21313 raft_consensus.cc:504] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.686547 21313 leader_election.cc:288] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:33.695627 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080b798481e34ca69b0df5141e439567" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946" is_pre_election: true
I20240503 02:21:33.695627 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080b798481e34ca69b0df5141e439567" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:33.695940 20838 raft_consensus.cc:2454] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:33.695956 20969 raft_consensus.cc:2454] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:33.696470 20640 leader_election.cc:302] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:33.696954 21313 raft_consensus.cc:2781] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:33.697115 21313 raft_consensus.cc:482] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:33.697223 21313 raft_consensus.cc:3037] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.699270 21313 raft_consensus.cc:504] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.699904 21313 leader_election.cc:288] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Requested vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:33.700461 20969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080b798481e34ca69b0df5141e439567" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:33.700464 20838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "080b798481e34ca69b0df5141e439567" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:33.700686 20838 raft_consensus.cc:3037] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.700697 20969 raft_consensus.cc:3037] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:33.702658 20838 raft_consensus.cc:2454] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:33.702663 20969 raft_consensus.cc:2454] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:33.703167 20642 leader_election.cc:302] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:33.703589 21313 raft_consensus.cc:2781] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:33.704083 21313 raft_consensus.cc:686] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Running, Role: LEADER
I20240503 02:21:33.704563 21313 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:33.709416 20564 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:33.871021 21309 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:33.878454 21333 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:34.103746 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:34.104146 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:34.111908 21328 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:21:34.113615 21306 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:34.119954 21322 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:34.127197 21313 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:34.132977 21326 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 325.643 ms
  time per row: 0.162822 ms
I20240503 02:21:34.601722 20565 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57184:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:34.602548 20565 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:34.611822 20685 tablet_service.cc:1467] Processing CreateTablet for tablet 33582788292045669f7ebb1eea5e0e05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.611822 20817 tablet_service.cc:1467] Processing CreateTablet for tablet 33582788292045669f7ebb1eea5e0e05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.612309 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33582788292045669f7ebb1eea5e0e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.612321 20685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33582788292045669f7ebb1eea5e0e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.613099 20949 tablet_service.cc:1467] Processing CreateTablet for tablet 33582788292045669f7ebb1eea5e0e05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.613549 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33582788292045669f7ebb1eea5e0e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.616578 20816 tablet_service.cc:1467] Processing CreateTablet for tablet b09ffd8e3f3148a6b1919682c19073a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.617065 20816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09ffd8e3f3148a6b1919682c19073a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.617689 20684 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde2d2192a041b78adbf768409e2d6d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.618137 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde2d2192a041b78adbf768409e2d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.621215 20947 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde2d2192a041b78adbf768409e2d6d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.621662 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde2d2192a041b78adbf768409e2d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.621708 20686 tablet_service.cc:1467] Processing CreateTablet for tablet b09ffd8e3f3148a6b1919682c19073a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.622079 20686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09ffd8e3f3148a6b1919682c19073a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.625753 20948 tablet_service.cc:1467] Processing CreateTablet for tablet b09ffd8e3f3148a6b1919682c19073a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.626153 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09ffd8e3f3148a6b1919682c19073a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.629498 20818 tablet_service.cc:1467] Processing CreateTablet for tablet 7cde2d2192a041b78adbf768409e2d6d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:34.630017 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde2d2192a041b78adbf768409e2d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:34.646272 21400 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:34.648206 21398 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:34.648377 21400 tablet_bootstrap.cc:654] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.650399 21398 tablet_bootstrap.cc:654] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.650830 21399 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:34.652729 21399 tablet_bootstrap.cc:654] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.652781 21398 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:34.652954 21398 ts_tablet_manager.cc:1367] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240503 02:21:34.652967 21400 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:34.653131 21400 ts_tablet_manager.cc:1367] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240503 02:21:34.653873 21398 raft_consensus.cc:348] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.653949 21400 raft_consensus.cc:348] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.654134 21398 raft_consensus.cc:374] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.654158 21400 raft_consensus.cc:374] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.654244 21400 raft_consensus.cc:729] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.654249 21398 raft_consensus.cc:729] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.654497 21398 consensus_queue.cc:260] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.654496 21400 consensus_queue.cc:260] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.655148 21398 ts_tablet_manager.cc:1398] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:21:34.655450 21399 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:34.655465 21398 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:34.655649 21399 ts_tablet_manager.cc:1367] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:21:34.655830 21400 ts_tablet_manager.cc:1398] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:21:34.656175 21400 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:34.656526 21399 raft_consensus.cc:348] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.656761 21399 raft_consensus.cc:374] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.656910 21399 raft_consensus.cc:729] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.657132 21398 tablet_bootstrap.cc:654] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.658468 21400 tablet_bootstrap.cc:654] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.657204 21399 consensus_queue.cc:260] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.659456 21399 ts_tablet_manager.cc:1398] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:21:34.659828 21399 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:34.661587 21399 tablet_bootstrap.cc:654] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.665591 21398 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:34.665916 21398 ts_tablet_manager.cc:1367] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240503 02:21:34.667091 21398 raft_consensus.cc:348] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.667472 21398 raft_consensus.cc:374] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.667557 21398 raft_consensus.cc:729] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.667819 21398 consensus_queue.cc:260] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.668478 21400 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:34.668637 21398 ts_tablet_manager.cc:1398] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:34.668713 21400 ts_tablet_manager.cc:1367] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.011s
I20240503 02:21:34.668987 21398 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:34.669601 21400 raft_consensus.cc:348] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.669772 21399 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:34.669924 21400 raft_consensus.cc:374] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.670002 21399 ts_tablet_manager.cc:1367] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.004s
I20240503 02:21:34.670058 21400 raft_consensus.cc:729] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.670400 21400 consensus_queue.cc:260] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.670809 21398 tablet_bootstrap.cc:654] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.670825 21399 raft_consensus.cc:348] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.671149 21399 raft_consensus.cc:374] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.671157 21400 ts_tablet_manager.cc:1398] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:21:34.671290 21399 raft_consensus.cc:729] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.671471 21400 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:34.671625 21399 consensus_queue.cc:260] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.672407 21399 ts_tablet_manager.cc:1398] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:34.672771 21399 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:34.673642 21400 tablet_bootstrap.cc:654] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.674712 21399 tablet_bootstrap.cc:654] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:34.679756 21398 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:34.679996 21398 ts_tablet_manager.cc:1367] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240503 02:21:34.681099 21398 raft_consensus.cc:348] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.681433 21398 raft_consensus.cc:374] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.681560 21398 raft_consensus.cc:729] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.681915 21398 consensus_queue.cc:260] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.682775 21398 ts_tablet_manager.cc:1398] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:34.683279 21399 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:34.683435 21399 ts_tablet_manager.cc:1367] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240503 02:21:34.684044 21400 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:34.684161 21400 ts_tablet_manager.cc:1367] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.005s
I20240503 02:21:34.684219 21399 raft_consensus.cc:348] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.684440 21399 raft_consensus.cc:374] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.684541 21399 raft_consensus.cc:729] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.684803 21399 consensus_queue.cc:260] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.684876 21400 raft_consensus.cc:348] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.685050 21400 raft_consensus.cc:374] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:34.685151 21400 raft_consensus.cc:729] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:34.685513 21399 ts_tablet_manager.cc:1398] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240503 02:21:34.685416 21400 consensus_queue.cc:260] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.686188 21400 ts_tablet_manager.cc:1398] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:34.712107 21322 raft_consensus.cc:482] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:34.712329 21322 raft_consensus.cc:504] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.713116 21322 leader_election.cc:288] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:34.713719 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33582788292045669f7ebb1eea5e0e05" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:34.713786 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33582788292045669f7ebb1eea5e0e05" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:34.713940 20705 raft_consensus.cc:2454] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:34.714061 20968 raft_consensus.cc:2454] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:34.714416 20774 leader_election.cc:302] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:34.714710 21322 raft_consensus.cc:2781] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:34.714824 21322 raft_consensus.cc:482] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:34.714948 21322 raft_consensus.cc:3037] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.718127 21322 raft_consensus.cc:504] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.718868 21322 leader_election.cc:288] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:34.719412 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33582788292045669f7ebb1eea5e0e05" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:34.719404 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33582788292045669f7ebb1eea5e0e05" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:34.719606 20705 raft_consensus.cc:3037] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.719614 20968 raft_consensus.cc:3037] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.721732 20968 raft_consensus.cc:2454] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:34.722227 20774 leader_election.cc:302] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:34.722523 21322 raft_consensus.cc:2781] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:34.722707 21322 raft_consensus.cc:686] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Leader. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:34.722697 20705 raft_consensus.cc:2454] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:34.722994 21322 consensus_queue.cc:237] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.725945 20564 catalog_manager.cc:5484] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7836d9d893434d7a99d61974b10dd946 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:34.735106 21333 raft_consensus.cc:482] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:34.735316 21333 raft_consensus.cc:504] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.736115 21333 leader_election.cc:288] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:34.736665 20836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde2d2192a041b78adbf768409e2d6d" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946" is_pre_election: true
I20240503 02:21:34.736712 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde2d2192a041b78adbf768409e2d6d" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:34.736961 20836 raft_consensus.cc:2454] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 0.
I20240503 02:21:34.736995 20705 raft_consensus.cc:2454] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 0.
I20240503 02:21:34.737418 20905 leader_election.cc:302] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:34.737730 21333 raft_consensus.cc:2781] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:34.737812 21333 raft_consensus.cc:482] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:34.737888 21333 raft_consensus.cc:3037] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.739784 21333 raft_consensus.cc:504] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.740386 21333 leader_election.cc:288] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 election: Requested vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:34.740916 20705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde2d2192a041b78adbf768409e2d6d" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:34.740958 20836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cde2d2192a041b78adbf768409e2d6d" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:34.741163 20836 raft_consensus.cc:3037] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.741175 20705 raft_consensus.cc:3037] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.743072 20705 raft_consensus.cc:2454] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 1.
I20240503 02:21:34.743166 20836 raft_consensus.cc:2454] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 1.
I20240503 02:21:34.743434 20905 leader_election.cc:302] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:34.743692 21333 raft_consensus.cc:2781] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:34.743875 21333 raft_consensus.cc:686] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Becoming Leader. State: Replica: c184c96b994d4707919e20333d152788, State: Running, Role: LEADER
I20240503 02:21:34.744151 21333 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.747145 20564 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: term changed from 0 to 1, leader changed from <none> to c184c96b994d4707919e20333d152788 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } }
I20240503 02:21:34.748713 21326 raft_consensus.cc:482] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:34.748911 21326 raft_consensus.cc:504] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.749774 21326 leader_election.cc:288] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:34.750360 20836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09ffd8e3f3148a6b1919682c19073a5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946" is_pre_election: true
I20240503 02:21:34.750540 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09ffd8e3f3148a6b1919682c19073a5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:34.750658 20836 raft_consensus.cc:2454] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:34.750802 20968 raft_consensus.cc:2454] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:34.751091 20640 leader_election.cc:302] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:34.751396 21326 raft_consensus.cc:2781] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:34.751483 21326 raft_consensus.cc:482] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:34.751585 21326 raft_consensus.cc:3037] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.753731 21326 raft_consensus.cc:504] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.754388 21326 leader_election.cc:288] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Requested vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:34.754772 20836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09ffd8e3f3148a6b1919682c19073a5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:34.754854 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09ffd8e3f3148a6b1919682c19073a5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:34.755013 20836 raft_consensus.cc:3037] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.755142 20968 raft_consensus.cc:3037] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:34.757019 20968 raft_consensus.cc:2454] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:34.757025 20836 raft_consensus.cc:2454] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:34.757442 20642 leader_election.cc:302] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:34.757777 21326 raft_consensus.cc:2781] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:34.757959 21326 raft_consensus.cc:686] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Running, Role: LEADER
I20240503 02:21:34.758239 21326 consensus_queue.cc:237] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:34.761174 20564 catalog_manager.cc:5484] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:35.141997 21326 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:35.148658 21326 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:35.152206 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:35.152650 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:35.160687 21413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:21:35.196439 21322 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:35.212035 21306 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:35.255657 21333 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:35.297686 21444 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:35.356544 20905 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.65:43579, user_credentials={real_user=slave}} blocked reactor thread for 66650.2us
W20240503 02:21:35.377629 20903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.218.66:33349, user_credentials={real_user=slave}} blocked reactor thread for 90361.7us

INSERT report
    rows total: 2000
    time total: 286.948 ms
  time per row: 0.143474 ms
I20240503 02:21:35.611613 20563 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57184:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:35.612421 20563 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:35.622224 20817 tablet_service.cc:1467] Processing CreateTablet for tablet 44d51a4af792486ab736338c2d377c67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.622133 20685 tablet_service.cc:1467] Processing CreateTablet for tablet 44d51a4af792486ab736338c2d377c67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.622694 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d51a4af792486ab736338c2d377c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.622730 20685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d51a4af792486ab736338c2d377c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.624186 20948 tablet_service.cc:1467] Processing CreateTablet for tablet 44d51a4af792486ab736338c2d377c67 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.624639 20948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d51a4af792486ab736338c2d377c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.625273 20818 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ffdf4cab040a6871acabfc0fec2e5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.625381 20816 tablet_service.cc:1467] Processing CreateTablet for tablet 1c431e0d457a44aaafff4abb86415214 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.625777 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1ffdf4cab040a6871acabfc0fec2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.629298 20947 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ffdf4cab040a6871acabfc0fec2e5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.629752 20947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1ffdf4cab040a6871acabfc0fec2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.631065 20816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c431e0d457a44aaafff4abb86415214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.633853 20949 tablet_service.cc:1467] Processing CreateTablet for tablet 1c431e0d457a44aaafff4abb86415214 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.634354 20949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c431e0d457a44aaafff4abb86415214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.637944 20686 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ffdf4cab040a6871acabfc0fec2e5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.638520 20686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1ffdf4cab040a6871acabfc0fec2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.640142 21481 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:35.642020 21481 tablet_bootstrap.cc:654] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.641896 20684 tablet_service.cc:1467] Processing CreateTablet for tablet 1c431e0d457a44aaafff4abb86415214 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:35.642591 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c431e0d457a44aaafff4abb86415214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:35.652742 21484 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:35.656714 21484 tablet_bootstrap.cc:654] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.659018 21481 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:35.659238 21481 ts_tablet_manager.cc:1367] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.014s
I20240503 02:21:35.659325 21482 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:35.660180 21481 raft_consensus.cc:348] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.660387 21481 raft_consensus.cc:374] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.660533 21481 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.660822 21481 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.661244 21484 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:35.661255 21482 tablet_bootstrap.cc:654] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.661432 21484 ts_tablet_manager.cc:1367] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240503 02:21:35.662286 21481 ts_tablet_manager.cc:1398] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240503 02:21:35.662251 21484 raft_consensus.cc:348] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.662451 21484 raft_consensus.cc:374] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.662568 21481 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:35.662539 21484 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.662828 21484 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.665112 21484 ts_tablet_manager.cc:1398] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:21:35.664186 21481 tablet_bootstrap.cc:654] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.668270 21484 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:35.670475 21484 tablet_bootstrap.cc:654] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.671131 21482 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:35.671209 21481 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:35.671299 21482 ts_tablet_manager.cc:1367] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240503 02:21:35.671391 21481 ts_tablet_manager.cc:1367] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 02:21:35.672230 21481 raft_consensus.cc:348] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.672441 21481 raft_consensus.cc:374] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.672530 21481 raft_consensus.cc:729] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.672459 21482 raft_consensus.cc:348] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.672840 21482 raft_consensus.cc:374] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.672776 21481 consensus_queue.cc:260] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.672959 21482 raft_consensus.cc:729] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.673349 21481 ts_tablet_manager.cc:1398] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:35.673223 21482 consensus_queue.cc:260] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.673707 21481 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:35.673991 21482 ts_tablet_manager.cc:1398] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:35.674242 21482 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:35.675349 21481 tablet_bootstrap.cc:654] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.676573 21482 tablet_bootstrap.cc:654] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.685364 21484 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:35.685596 21484 ts_tablet_manager.cc:1367] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.011s
I20240503 02:21:35.685628 21481 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: No bootstrap required, opened a new log
I20240503 02:21:35.685782 21481 ts_tablet_manager.cc:1367] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.012s	user 0.008s	sys 0.004s
I20240503 02:21:35.686183 21482 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:35.686364 21482 ts_tablet_manager.cc:1367] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.009s
I20240503 02:21:35.686311 21484 raft_consensus.cc:348] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.686563 21484 raft_consensus.cc:374] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.686641 21484 raft_consensus.cc:729] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.686614 21481 raft_consensus.cc:348] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.686811 21481 raft_consensus.cc:374] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.686910 21481 raft_consensus.cc:729] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.686934 21484 consensus_queue.cc:260] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.687140 21481 consensus_queue.cc:260] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.687546 21484 ts_tablet_manager.cc:1398] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:21:35.687703 21481 ts_tablet_manager.cc:1398] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:35.687849 21484 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:35.687170 21482 raft_consensus.cc:348] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.688580 21482 raft_consensus.cc:374] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.688665 21482 raft_consensus.cc:729] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.688923 21482 consensus_queue.cc:260] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.689550 21482 ts_tablet_manager.cc:1398] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240503 02:21:35.689957 21482 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:35.690186 21484 tablet_bootstrap.cc:654] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.691779 21482 tablet_bootstrap.cc:654] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:35.693969 21482 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: No bootstrap required, opened a new log
I20240503 02:21:35.694118 21482 ts_tablet_manager.cc:1367] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:35.694312 21484 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788: No bootstrap required, opened a new log
I20240503 02:21:35.694456 21484 ts_tablet_manager.cc:1367] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240503 02:21:35.695047 21482 raft_consensus.cc:348] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.695107 21484 raft_consensus.cc:348] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.695262 21484 raft_consensus.cc:374] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.695268 21482 raft_consensus.cc:374] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:35.695353 21484 raft_consensus.cc:729] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.695367 21482 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: FOLLOWER
I20240503 02:21:35.695609 21484 consensus_queue.cc:260] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.695595 21482 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.696103 21482 ts_tablet_manager.cc:1398] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:35.696305 21484 ts_tablet_manager.cc:1398] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:35.850798 21322 raft_consensus.cc:482] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:35.851076 21322 raft_consensus.cc:504] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.851783 21322 leader_election.cc:288] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.852458 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:35.852511 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:35.852768 20706 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:35.852779 20968 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:35.853288 20774 leader_election.cc:302] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:35.853701 21322 raft_consensus.cc:2781] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:35.853788 21322 raft_consensus.cc:482] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:35.853933 21322 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.855978 21322 raft_consensus.cc:504] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.856710 21322 leader_election.cc:288] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.857184 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:35.857219 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:35.857429 20706 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.857450 20968 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.859277 20968 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:35.859526 20706 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:35.859719 20774 leader_election.cc:302] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:35.860034 21322 raft_consensus.cc:2781] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:35.860210 21322 raft_consensus.cc:686] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Leader. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:35.860496 21322 consensus_queue.cc:237] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.863178 20560 catalog_manager.cc:5484] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7836d9d893434d7a99d61974b10dd946 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:35.956529 21383 raft_consensus.cc:482] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:35.956729 21383 raft_consensus.cc:504] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.957518 21383 leader_election.cc:288] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.958107 20837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1ffdf4cab040a6871acabfc0fec2e5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946" is_pre_election: true
I20240503 02:21:35.958158 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1ffdf4cab040a6871acabfc0fec2e5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:35.958372 20837 raft_consensus.cc:2454] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:35.958390 20968 raft_consensus.cc:2454] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 0.
I20240503 02:21:35.958868 20640 leader_election.cc:302] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:35.959157 21383 raft_consensus.cc:2781] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:35.959275 21383 raft_consensus.cc:482] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:35.959378 21383 raft_consensus.cc:3037] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.961525 21383 raft_consensus.cc:504] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.962203 21383 leader_election.cc:288] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Requested vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.962647 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1ffdf4cab040a6871acabfc0fec2e5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:35.962642 20837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1ffdf4cab040a6871acabfc0fec2e5" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:35.962854 20837 raft_consensus.cc:3037] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.962857 20968 raft_consensus.cc:3037] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.965742 20837 raft_consensus.cc:2454] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:35.965736 20968 raft_consensus.cc:2454] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 1.
I20240503 02:21:35.966210 20640 leader_election.cc:302] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:35.966529 21383 raft_consensus.cc:2781] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:35.966646 21383 raft_consensus.cc:686] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Running, Role: LEADER
I20240503 02:21:35.966902 21383 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.969559 20560 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65). New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:35.991740 21322 raft_consensus.cc:482] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:35.991955 21322 raft_consensus.cc:504] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.992602 21322 leader_election.cc:288] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.993196 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3" is_pre_election: true
I20240503 02:21:35.993312 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788" is_pre_election: true
I20240503 02:21:35.993474 20706 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:35.993599 20968 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 0.
I20240503 02:21:35.993916 20774 leader_election.cc:302] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba5197bbdb144878029533fd45d53b3, 7836d9d893434d7a99d61974b10dd946; no voters: 
I20240503 02:21:35.994232 21322 raft_consensus.cc:2781] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:35.994354 21322 raft_consensus.cc:482] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:35.994444 21322 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.996373 21322 raft_consensus.cc:504] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:35.997136 21322 leader_election.cc:288] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:35.997531 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:35.997622 20968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "7836d9d893434d7a99d61974b10dd946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:35.997773 20706 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.997835 20968 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:35.999634 20706 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:35.999675 20968 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7836d9d893434d7a99d61974b10dd946 in term 1.
I20240503 02:21:36.000077 20774 leader_election.cc:302] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7836d9d893434d7a99d61974b10dd946, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:36.000383 21322 raft_consensus.cc:2781] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:36.000528 21322 raft_consensus.cc:686] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Leader. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:36.000841 21322 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } }
I20240503 02:21:36.003531 20560 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7836d9d893434d7a99d61974b10dd946 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:36.255762 21322 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:36.268651 21459 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:36.372565 21383 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:36.374742 21383 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:36.460034 21496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:36.460429 21496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:36.468246 21496 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:21:36.541065 21459 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:36.549772 21322 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 244.235 ms
  time per row: 0.122118 ms
I20240503 02:21:36.868891 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.69:44045
--local_ip_for_outbound_sockets=127.20.218.69
--tserver_master_addrs=127.20.218.126:38245
--webserver_port=37781
--webserver_interface=127.20.218.69
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:21:37.238035 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:37.238350 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:37.238497 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:37.247380 21554 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:37.247589 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:37.247786 21554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.69
I20240503 02:21:37.258289 21554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.69:44045
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.218.69
--webserver_port=37781
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.69
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:37.259706 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:37.261628 21554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:37.274907 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:37.274835 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:37.275310 21554 server_base.cc:1034] running on GCE node
W20240503 02:21:37.274873 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:37.276214 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:37.277199 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:37.278435 21554 hybrid_clock.cc:648] HybridClock initialized: now 1714702897278377 us; error 58 us; skew 500 ppm
I20240503 02:21:37.278784 21554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:37.281131 21554 webserver.cc:460] Webserver started at http://127.20.218.69:37781/ using document root <none> and password file <none>
I20240503 02:21:37.281915 21554 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:37.282069 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:37.288717 21554 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240503 02:21:37.292032 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.293525 21554 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:37.293712 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "46b0cc47f0274f63a284f07b49cc6da5"
format_stamp: "Formatted at 2024-05-03 02:21:29 on dist-test-slave-81lq"
I20240503 02:21:37.294587 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:37.343073 21554 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:37.343993 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:37.344331 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:37.345443 21554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:37.347232 21554 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:37.347322 21554 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.347453 21554 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:37.347510 21554 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.388546 21554 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.69:44045
I20240503 02:21:37.388602 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.69:44045 every 8 connection(s)
I20240503 02:21:37.390087 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:21:37.395929 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21554
I20240503 02:21:37.396732 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:36887
--local_ip_for_outbound_sockets=127.20.218.68
--tserver_master_addrs=127.20.218.126:38245
--webserver_port=39727
--webserver_interface=127.20.218.68
--builtin_ntp_servers=127.20.218.84:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:37.405858 21681 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:37.406224 21681 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:37.407157 21681 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:37.408905 20557 ts_manager.cc:194] Re-registered known tserver with Master: 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045)
I20240503 02:21:37.409891 20557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.69:59661
W20240503 02:21:37.802067 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:37.802687 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:37.802943 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:37.812747 21685 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:37.812918 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:37.813148 21685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:37.823196 21685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:36887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=39727
--tserver_master_addrs=127.20.218.126:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:37.824710 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:37.826557 21685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:37.840390 21694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:37.840515 21691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:37.840502 21692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:37.841509 21685 server_base.cc:1034] running on GCE node
I20240503 02:21:37.842217 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:37.843405 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:37.844692 21685 hybrid_clock.cc:648] HybridClock initialized: now 1714702897844620 us; error 80 us; skew 500 ppm
I20240503 02:21:37.845139 21685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:37.854934 21685 webserver.cc:460] Webserver started at http://127.20.218.68:39727/ using document root <none> and password file <none>
I20240503 02:21:37.855901 21685 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:37.856070 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:37.862335 21685 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240503 02:21:37.865764 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.867249 21685 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240503 02:21:37.867412 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "16c046bf08ee4bd684ff0fcb5b158d4d"
format_stamp: "Formatted at 2024-05-03 02:21:28 on dist-test-slave-81lq"
I20240503 02:21:37.868263 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:37.900606 21685 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:37.901575 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:37.901991 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:37.903084 21685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:37.904915 21685 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:37.905005 21685 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.905141 21685 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:37.905200 21685 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:37.941053 21685 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:36887
I20240503 02:21:37.941077 21812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:36887 every 8 connection(s)
I20240503 02:21:37.942667 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:37.949432 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 21685
I20240503 02:21:37.956794 21813 heartbeater.cc:347] Connected to a master server at 127.20.218.126:38245
I20240503 02:21:37.957146 21813 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:37.958159 21813 heartbeater.cc:511] Master 127.20.218.126:38245 requested a full tablet report, sending...
I20240503 02:21:37.959910 20557 ts_manager.cc:194] Re-registered known tserver with Master: 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887)
I20240503 02:21:37.961202 20557 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:38513
I20240503 02:21:38.413012 21681 heartbeater.cc:503] Master 127.20.218.126:38245 was elected leader, sending a full tablet report...
I20240503 02:21:38.520475 20686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:38.520818 20818 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:38.524359 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.527925 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.531767 20949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:38.735827 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.742750 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.747277 20686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:38.760229 20818 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:38.776115 20949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:38.938624 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.939435 20686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:38.945174 20818 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:38.945930 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:38.951756 20949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:38.964442 21813 heartbeater.cc:503] Master 127.20.218.126:38245 was elected leader, sending a full tablet report...
I20240503 02:21:39.126354 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:39.127545 20686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:39.135552 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:39.141533 20818 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:39.145093 20949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:39.238813 20837 consensus_queue.cc:237] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.241962 20705 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.242092 20968 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.242810 21322 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.243937 21960 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.254799 21322 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
W20240503 02:21:39.255826 20771 consensus_peers.cc:485] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 -> Peer 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887): Couldn't send request to peer 16c046bf08ee4bd684ff0fcb5b158d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33582788292045669f7ebb1eea5e0e05. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.259881 20968 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.261564 20705 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.263883 20564 catalog_manager.cc:5484] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:39.279639 20705 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:39.282770 20968 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.282836 20837 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.283653 21974 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.284077 21689 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.287636 21689 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.288125 20968 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
W20240503 02:21:39.292676 20639 consensus_peers.cc:485] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 -> Peer 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045): Couldn't send request to peer 46b0cc47f0274f63a284f07b49cc6da5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba1ffdf4cab040a6871acabfc0fec2e5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.293871 20556 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:39.296547 20837 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.307731 20837 consensus_queue.cc:237] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.310439 20705 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.310458 20968 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:21:39.311085 20771 consensus_peers.cc:485] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 -> Peer 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887): Couldn't send request to peer 16c046bf08ee4bd684ff0fcb5b158d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c2e64b53d804601b7311747a3f8dce4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.311087 21322 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.311573 21901 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.314633 21960 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.315503 20705 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.315515 20968 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.320386 20557 catalog_manager.cc:5484] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.339761 20968 consensus_queue.cc:237] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:39.342898 20837 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.342936 20705 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.343611 21333 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.343991 21993 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.347761 21333 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
W20240503 02:21:39.351179 20902 consensus_peers.cc:485] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 -> Peer 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045): Couldn't send request to peer 46b0cc47f0274f63a284f07b49cc6da5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a69bdcc01c674a81a3cfd83e9eaaea94. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.354085 20837 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.355842 20705 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.359908 20564 catalog_manager.cc:5484] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.374369 20705 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.376933 20837 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.376936 20968 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.377632 21974 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.378090 21978 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.382138 21974 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.385950 20967 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
W20240503 02:21:39.390269 20639 consensus_peers.cc:485] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 -> Peer 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887): Couldn't send request to peer 16c046bf08ee4bd684ff0fcb5b158d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 080b798481e34ca69b0df5141e439567. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.391453 20557 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.397924 20837 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.414122 20967 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.417186 20705 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.417989 20837 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.418510 22011 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.418951 21993 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.421833 21993 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
W20240503 02:21:39.425413 20902 consensus_peers.cc:485] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 -> Peer 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887): Couldn't send request to peer 16c046bf08ee4bd684ff0fcb5b158d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cde2d2192a041b78adbf768409e2d6d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.425446 20705 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.427692 20557 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:39.427692 20837 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:39.445581 20705 consensus_queue.cc:237] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:39.448285 20837 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:39.448313 20967 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:21:39.448818 20639 consensus_peers.cc:485] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 -> Peer 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045): Couldn't send request to peer 46b0cc47f0274f63a284f07b49cc6da5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b09ffd8e3f3148a6b1919682c19073a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:39.448925 21974 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.449353 21689 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:39.452929 21978 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.453891 20967 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.454020 20837 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:39.458424 20557 catalog_manager.cc:5484] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:39.717464 22057 ts_tablet_manager.cc:902] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Initiating tablet copy from peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:39.724282 22062 ts_tablet_manager.cc:902] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Initiating tablet copy from peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:39.730139 22057 tablet_copy_client.cc:308] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:33349
I20240503 02:21:39.731685 22062 tablet_copy_client.cc:308] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:33349
I20240503 02:21:39.739212 20858 tablet_copy_service.cc:140] P 7836d9d893434d7a99d61974b10dd946: Received BeginTabletCopySession request for tablet 9c2e64b53d804601b7311747a3f8dce4 from peer 16c046bf08ee4bd684ff0fcb5b158d4d ({username='slave'} at 127.20.218.68:50591)
I20240503 02:21:39.739187 20857 tablet_copy_service.cc:140] P 7836d9d893434d7a99d61974b10dd946: Received BeginTabletCopySession request for tablet 33582788292045669f7ebb1eea5e0e05 from peer 16c046bf08ee4bd684ff0fcb5b158d4d ({username='slave'} at 127.20.218.68:50591)
I20240503 02:21:39.739604 20858 tablet_copy_service.cc:161] P 7836d9d893434d7a99d61974b10dd946: Beginning new tablet copy session on tablet 9c2e64b53d804601b7311747a3f8dce4 from peer 16c046bf08ee4bd684ff0fcb5b158d4d at {username='slave'} at 127.20.218.68:50591: session id = 16c046bf08ee4bd684ff0fcb5b158d4d-9c2e64b53d804601b7311747a3f8dce4
I20240503 02:21:39.740118 20857 tablet_copy_service.cc:161] P 7836d9d893434d7a99d61974b10dd946: Beginning new tablet copy session on tablet 33582788292045669f7ebb1eea5e0e05 from peer 16c046bf08ee4bd684ff0fcb5b158d4d at {username='slave'} at 127.20.218.68:50591: session id = 16c046bf08ee4bd684ff0fcb5b158d4d-33582788292045669f7ebb1eea5e0e05
I20240503 02:21:39.744015 20858 tablet_copy_source_session.cc:205] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:39.746147 20857 tablet_copy_source_session.cc:205] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:39.750517 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c2e64b53d804601b7311747a3f8dce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:39.753896 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33582788292045669f7ebb1eea5e0e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:39.766913 22062 tablet_copy_client.cc:783] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:39.767287 22057 tablet_copy_client.cc:783] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:39.767321 22062 tablet_copy_client.cc:647] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:39.767570 22057 tablet_copy_client.cc:647] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:39.770264 22062 tablet_copy_client.cc:523] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:39.770690 22057 tablet_copy_client.cc:523] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:39.775295 22057 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap starting.
I20240503 02:21:39.776633 22062 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap starting.
I20240503 02:21:39.779898 22082 ts_tablet_manager.cc:902] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Initiating tablet copy from peer c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:39.794458 22057 log.cc:824] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:39.829911 22082 tablet_copy_client.cc:308] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41081
I20240503 02:21:39.856566 20989 tablet_copy_service.cc:140] P c184c96b994d4707919e20333d152788: Received BeginTabletCopySession request for tablet a69bdcc01c674a81a3cfd83e9eaaea94 from peer 46b0cc47f0274f63a284f07b49cc6da5 ({username='slave'} at 127.20.218.69:57099)
I20240503 02:21:39.857012 20989 tablet_copy_service.cc:161] P c184c96b994d4707919e20333d152788: Beginning new tablet copy session on tablet a69bdcc01c674a81a3cfd83e9eaaea94 from peer 46b0cc47f0274f63a284f07b49cc6da5 at {username='slave'} at 127.20.218.69:57099: session id = 46b0cc47f0274f63a284f07b49cc6da5-a69bdcc01c674a81a3cfd83e9eaaea94
I20240503 02:21:39.859098 22098 ts_tablet_manager.cc:902] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Initiating tablet copy from peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:39.860772 22099 ts_tablet_manager.cc:902] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Initiating tablet copy from peer c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:39.861765 20989 tablet_copy_source_session.cc:205] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:39.867735 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a69bdcc01c674a81a3cfd83e9eaaea94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:39.885115 22057 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:39.886081 22057 tablet_bootstrap.cc:492] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap complete.
I20240503 02:21:39.887516 22057 ts_tablet_manager.cc:1367] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent bootstrapping tablet: real 0.112s	user 0.071s	sys 0.026s
I20240503 02:21:39.891912 22082 tablet_copy_client.cc:783] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:39.892385 22082 tablet_copy_client.cc:647] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:39.895654 22082 tablet_copy_client.cc:523] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:39.899994 22057 raft_consensus.cc:348] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.901690 22082 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap starting.
I20240503 02:21:39.903537 22099 tablet_copy_client.cc:308] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41081
I20240503 02:21:39.903965 22057 raft_consensus.cc:729] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c046bf08ee4bd684ff0fcb5b158d4d, State: Initialized, Role: LEARNER
I20240503 02:21:39.904780 22057 consensus_queue.cc:260] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.906858 22062 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:39.907366 22062 tablet_bootstrap.cc:492] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap complete.
I20240503 02:21:39.907687 22062 ts_tablet_manager.cc:1367] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent bootstrapping tablet: real 0.131s	user 0.081s	sys 0.011s
I20240503 02:21:39.908751 22062 raft_consensus.cc:348] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.909085 22062 raft_consensus.cc:729] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c046bf08ee4bd684ff0fcb5b158d4d, State: Initialized, Role: LEARNER
I20240503 02:21:39.912926 22111 ts_tablet_manager.cc:902] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Initiating tablet copy from peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:39.913668 22062 consensus_queue.cc:260] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:39.915268 22098 tablet_copy_client.cc:308] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:43579
I20240503 02:21:39.915745 20989 tablet_copy_service.cc:140] P c184c96b994d4707919e20333d152788: Received BeginTabletCopySession request for tablet 7cde2d2192a041b78adbf768409e2d6d from peer 16c046bf08ee4bd684ff0fcb5b158d4d ({username='slave'} at 127.20.218.68:36051)
I20240503 02:21:39.916009 20989 tablet_copy_service.cc:161] P c184c96b994d4707919e20333d152788: Beginning new tablet copy session on tablet 7cde2d2192a041b78adbf768409e2d6d from peer 16c046bf08ee4bd684ff0fcb5b158d4d at {username='slave'} at 127.20.218.68:36051: session id = 16c046bf08ee4bd684ff0fcb5b158d4d-7cde2d2192a041b78adbf768409e2d6d
I20240503 02:21:39.921036 20989 tablet_copy_source_session.cc:205] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:39.932889 22062 ts_tablet_manager.cc:1398] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.004s
I20240503 02:21:39.937151 22057 ts_tablet_manager.cc:1398] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent starting tablet: real 0.049s	user 0.029s	sys 0.012s
I20240503 02:21:39.938279 20857 tablet_copy_service.cc:342] P 7836d9d893434d7a99d61974b10dd946: Request end of tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-33582788292045669f7ebb1eea5e0e05 received from {username='slave'} at 127.20.218.68:50591
I20240503 02:21:39.938359 22099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde2d2192a041b78adbf768409e2d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:39.938474 20857 tablet_copy_service.cc:433] P 7836d9d893434d7a99d61974b10dd946: ending tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-33582788292045669f7ebb1eea5e0e05 on tablet 33582788292045669f7ebb1eea5e0e05 with peer 16c046bf08ee4bd684ff0fcb5b158d4d
I20240503 02:21:39.939189 20858 tablet_copy_service.cc:342] P 7836d9d893434d7a99d61974b10dd946: Request end of tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-9c2e64b53d804601b7311747a3f8dce4 received from {username='slave'} at 127.20.218.68:50591
I20240503 02:21:39.939324 20858 tablet_copy_service.cc:433] P 7836d9d893434d7a99d61974b10dd946: ending tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-9c2e64b53d804601b7311747a3f8dce4 on tablet 9c2e64b53d804601b7311747a3f8dce4 with peer 16c046bf08ee4bd684ff0fcb5b158d4d
I20240503 02:21:39.946909 20726 tablet_copy_service.cc:140] P 5ba5197bbdb144878029533fd45d53b3: Received BeginTabletCopySession request for tablet 080b798481e34ca69b0df5141e439567 from peer 16c046bf08ee4bd684ff0fcb5b158d4d ({username='slave'} at 127.20.218.68:48183)
I20240503 02:21:39.947189 20726 tablet_copy_service.cc:161] P 5ba5197bbdb144878029533fd45d53b3: Beginning new tablet copy session on tablet 080b798481e34ca69b0df5141e439567 from peer 16c046bf08ee4bd684ff0fcb5b158d4d at {username='slave'} at 127.20.218.68:48183: session id = 16c046bf08ee4bd684ff0fcb5b158d4d-080b798481e34ca69b0df5141e439567
I20240503 02:21:39.951058 20726 tablet_copy_source_session.cc:205] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:39.953466 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080b798481e34ca69b0df5141e439567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:39.965966 22082 log.cc:824] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:39.966593 22099 tablet_copy_client.cc:783] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:39.967054 22099 tablet_copy_client.cc:647] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:39.969588 22099 tablet_copy_client.cc:523] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:39.969841 22098 tablet_copy_client.cc:783] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:39.970352 22098 tablet_copy_client.cc:647] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:39.976585 22098 tablet_copy_client.cc:523] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:39.981395 22098 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap starting.
I20240503 02:21:39.985836 22099 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap starting.
I20240503 02:21:39.989166 22125 ts_tablet_manager.cc:902] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Initiating tablet copy from peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:39.993849 22111 tablet_copy_client.cc:308] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:43579
I20240503 02:21:40.005955 20726 tablet_copy_service.cc:140] P 5ba5197bbdb144878029533fd45d53b3: Received BeginTabletCopySession request for tablet ba1ffdf4cab040a6871acabfc0fec2e5 from peer 46b0cc47f0274f63a284f07b49cc6da5 ({username='slave'} at 127.20.218.69:53803)
I20240503 02:21:40.006261 20726 tablet_copy_service.cc:161] P 5ba5197bbdb144878029533fd45d53b3: Beginning new tablet copy session on tablet ba1ffdf4cab040a6871acabfc0fec2e5 from peer 46b0cc47f0274f63a284f07b49cc6da5 at {username='slave'} at 127.20.218.69:53803: session id = 46b0cc47f0274f63a284f07b49cc6da5-ba1ffdf4cab040a6871acabfc0fec2e5
I20240503 02:21:40.011222 20726 tablet_copy_source_session.cc:205] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:40.013835 22111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1ffdf4cab040a6871acabfc0fec2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:40.031396 22111 tablet_copy_client.cc:783] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:40.032016 22111 tablet_copy_client.cc:647] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:40.034624 22111 tablet_copy_client.cc:523] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:40.050853 22111 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap starting.
I20240503 02:21:40.059868 22125 tablet_copy_client.cc:308] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:43579
I20240503 02:21:40.060847 20726 tablet_copy_service.cc:140] P 5ba5197bbdb144878029533fd45d53b3: Received BeginTabletCopySession request for tablet b09ffd8e3f3148a6b1919682c19073a5 from peer 46b0cc47f0274f63a284f07b49cc6da5 ({username='slave'} at 127.20.218.69:53803)
I20240503 02:21:40.061044 20726 tablet_copy_service.cc:161] P 5ba5197bbdb144878029533fd45d53b3: Beginning new tablet copy session on tablet b09ffd8e3f3148a6b1919682c19073a5 from peer 46b0cc47f0274f63a284f07b49cc6da5 at {username='slave'} at 127.20.218.69:53803: session id = 46b0cc47f0274f63a284f07b49cc6da5-b09ffd8e3f3148a6b1919682c19073a5
I20240503 02:21:40.064229 20726 tablet_copy_source_session.cc:205] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:40.065855 22125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09ffd8e3f3148a6b1919682c19073a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:40.079794 22125 tablet_copy_client.cc:783] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:40.080197 22125 tablet_copy_client.cc:647] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:40.082350 22125 tablet_copy_client.cc:523] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:40.082489 22099 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:40.083047 22099 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap complete.
I20240503 02:21:40.083415 22099 ts_tablet_manager.cc:1367] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent bootstrapping tablet: real 0.098s	user 0.066s	sys 0.027s
I20240503 02:21:40.084376 22099 raft_consensus.cc:348] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:40.084725 22099 raft_consensus.cc:729] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c046bf08ee4bd684ff0fcb5b158d4d, State: Initialized, Role: LEARNER
I20240503 02:21:40.084967 22099 consensus_queue.cc:260] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:40.085748 22099 ts_tablet_manager.cc:1398] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240503 02:21:40.086743 20989 tablet_copy_service.cc:342] P c184c96b994d4707919e20333d152788: Request end of tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-7cde2d2192a041b78adbf768409e2d6d received from {username='slave'} at 127.20.218.68:36051
I20240503 02:21:40.086920 20989 tablet_copy_service.cc:433] P c184c96b994d4707919e20333d152788: ending tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-7cde2d2192a041b78adbf768409e2d6d on tablet 7cde2d2192a041b78adbf768409e2d6d with peer 16c046bf08ee4bd684ff0fcb5b158d4d
I20240503 02:21:40.087769 22125 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap starting.
I20240503 02:21:40.097918 22098 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:40.098840 22098 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap complete.
I20240503 02:21:40.099238 22098 ts_tablet_manager.cc:1367] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent bootstrapping tablet: real 0.118s	user 0.065s	sys 0.010s
I20240503 02:21:40.100261 22098 raft_consensus.cc:348] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:40.100590 22098 raft_consensus.cc:729] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c046bf08ee4bd684ff0fcb5b158d4d, State: Initialized, Role: LEARNER
I20240503 02:21:40.100950 22098 consensus_queue.cc:260] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:40.101706 22098 ts_tablet_manager.cc:1398] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240503 02:21:40.103233 20726 tablet_copy_service.cc:342] P 5ba5197bbdb144878029533fd45d53b3: Request end of tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-080b798481e34ca69b0df5141e439567 received from {username='slave'} at 127.20.218.68:48183
I20240503 02:21:40.103474 20726 tablet_copy_service.cc:433] P 5ba5197bbdb144878029533fd45d53b3: ending tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-080b798481e34ca69b0df5141e439567 on tablet 080b798481e34ca69b0df5141e439567 with peer 16c046bf08ee4bd684ff0fcb5b158d4d
I20240503 02:21:40.108521 22082 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:40.109527 22082 tablet_bootstrap.cc:492] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap complete.
I20240503 02:21:40.110139 22082 ts_tablet_manager.cc:1367] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent bootstrapping tablet: real 0.209s	user 0.070s	sys 0.012s
I20240503 02:21:40.118113 22082 raft_consensus.cc:348] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.118762 22082 raft_consensus.cc:729] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46b0cc47f0274f63a284f07b49cc6da5, State: Initialized, Role: LEARNER
I20240503 02:21:40.119467 22082 consensus_queue.cc:260] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.121728 22082 ts_tablet_manager.cc:1398] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.001s
I20240503 02:21:40.122985 20989 tablet_copy_service.cc:342] P c184c96b994d4707919e20333d152788: Request end of tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-a69bdcc01c674a81a3cfd83e9eaaea94 received from {username='slave'} at 127.20.218.69:57099
I20240503 02:21:40.123189 20989 tablet_copy_service.cc:433] P c184c96b994d4707919e20333d152788: ending tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-a69bdcc01c674a81a3cfd83e9eaaea94 on tablet a69bdcc01c674a81a3cfd83e9eaaea94 with peer 46b0cc47f0274f63a284f07b49cc6da5
I20240503 02:21:40.130919 22111 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:40.131458 22111 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap complete.
I20240503 02:21:40.131803 22111 ts_tablet_manager.cc:1367] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent bootstrapping tablet: real 0.081s	user 0.041s	sys 0.011s
I20240503 02:21:40.133816 21767 raft_consensus.cc:1203] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.134013 22111 raft_consensus.cc:348] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.134287 22111 raft_consensus.cc:729] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46b0cc47f0274f63a284f07b49cc6da5, State: Initialized, Role: LEARNER
I20240503 02:21:40.134526 22111 consensus_queue.cc:260] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.135731 22111 ts_tablet_manager.cc:1398] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:21:40.137441 20726 tablet_copy_service.cc:342] P 5ba5197bbdb144878029533fd45d53b3: Request end of tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-ba1ffdf4cab040a6871acabfc0fec2e5 received from {username='slave'} at 127.20.218.69:53803
I20240503 02:21:40.137696 20726 tablet_copy_service.cc:433] P 5ba5197bbdb144878029533fd45d53b3: ending tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-ba1ffdf4cab040a6871acabfc0fec2e5 on tablet ba1ffdf4cab040a6871acabfc0fec2e5 with peer 46b0cc47f0274f63a284f07b49cc6da5
I20240503 02:21:40.146785 22125 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:40.147213 22125 tablet_bootstrap.cc:492] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap complete.
I20240503 02:21:40.147491 22125 ts_tablet_manager.cc:1367] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent bootstrapping tablet: real 0.060s	user 0.048s	sys 0.008s
I20240503 02:21:40.148155 22125 raft_consensus.cc:348] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.148380 22125 raft_consensus.cc:729] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46b0cc47f0274f63a284f07b49cc6da5, State: Initialized, Role: LEARNER
I20240503 02:21:40.148572 22125 consensus_queue.cc:260] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:40.149044 22125 ts_tablet_manager.cc:1398] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:21:40.150270 20726 tablet_copy_service.cc:342] P 5ba5197bbdb144878029533fd45d53b3: Request end of tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-b09ffd8e3f3148a6b1919682c19073a5 received from {username='slave'} at 127.20.218.69:53803
I20240503 02:21:40.150621 20726 tablet_copy_service.cc:433] P 5ba5197bbdb144878029533fd45d53b3: ending tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-b09ffd8e3f3148a6b1919682c19073a5 on tablet b09ffd8e3f3148a6b1919682c19073a5 with peer 46b0cc47f0274f63a284f07b49cc6da5
I20240503 02:21:40.220736 21767 raft_consensus.cc:1203] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.324901 21636 raft_consensus.cc:1203] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.372615 21767 raft_consensus.cc:1203] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.406327 21636 raft_consensus.cc:1203] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.418337 21767 raft_consensus.cc:1203] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.477361 21636 raft_consensus.cc:1203] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:40.555675 22140 raft_consensus.cc:1050] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946: attempting to promote NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d to VOTER
I20240503 02:21:40.556813 22140 consensus_queue.cc:237] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.561028 21767 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.561419 20968 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.561793 22140 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.562371 20705 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.562886 22140 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.563196 22141 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.566802 21960 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.567487 20968 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.571359 20705 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.574286 21767 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.574368 20557 catalog_manager.cc:5484] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.589696 20837 consensus_queue.cc:237] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.591811 21767 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.592249 20968 raft_consensus.cc:1261] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.592526 22190 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.592902 22140 consensus_queue.cc:1035] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.595618 22141 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.596501 21767 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.599872 20550 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 33582788292045669f7ebb1eea5e0e05 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:40.601413 20556 catalog_manager.cc:5484] T 33582788292045669f7ebb1eea5e0e05 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:40.605636 20686 tablet_service.cc:1514] Processing DeleteTablet for tablet 33582788292045669f7ebb1eea5e0e05 with delete_type TABLET_DATA_TOMBSTONED (TS 5ba5197bbdb144878029533fd45d53b3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48912
I20240503 02:21:40.606041 20968 raft_consensus.cc:2932] T 33582788292045669f7ebb1eea5e0e05 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.617977 22200 tablet_replica.cc:330] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: stopping tablet replica
I20240503 02:21:40.618961 22200 raft_consensus.cc:2229] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:40.619355 22200 raft_consensus.cc:2258] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:40.622035 22200 ts_tablet_manager.cc:1875] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:40.627727 22200 ts_tablet_manager.cc:1888] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:40.627871 22200 log.cc:1196] T 33582788292045669f7ebb1eea5e0e05 P 5ba5197bbdb144878029533fd45d53b3: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/33582788292045669f7ebb1eea5e0e05
I20240503 02:21:40.628849 20552 catalog_manager.cc:4830] TS 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): tablet 33582788292045669f7ebb1eea5e0e05 (table rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]) successfully deleted
I20240503 02:21:40.744503 21960 raft_consensus.cc:1050] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946: attempting to promote NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d to VOTER
I20240503 02:21:40.745314 21960 consensus_queue.cc:237] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.747712 21767 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.747759 20968 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.747788 20705 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.748308 22140 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.748646 21960 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.748917 22141 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.752418 21960 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.753137 20968 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.753193 21767 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.757562 20705 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.757838 20556 catalog_manager.cc:5484] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.764276 20837 consensus_queue.cc:237] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.766202 21767 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.766429 20705 raft_consensus.cc:1261] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.766749 21960 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.767045 22141 consensus_queue.cc:1035] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.769409 22141 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.770248 20705 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.770545 21767 raft_consensus.cc:2932] T 9c2e64b53d804601b7311747a3f8dce4 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.772568 20550 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9c2e64b53d804601b7311747a3f8dce4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:40.774686 20565 catalog_manager.cc:5484] T 9c2e64b53d804601b7311747a3f8dce4 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:40.778921 20949 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c2e64b53d804601b7311747a3f8dce4 with delete_type TABLET_DATA_TOMBSTONED (TS c184c96b994d4707919e20333d152788 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35000
I20240503 02:21:40.779975 22260 tablet_replica.cc:330] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: stopping tablet replica
I20240503 02:21:40.780400 22260 raft_consensus.cc:2229] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:40.780751 22260 raft_consensus.cc:2258] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:40.782327 22260 ts_tablet_manager.cc:1875] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:40.788182 22260 ts_tablet_manager.cc:1888] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:40.788357 22260 log.cc:1196] T 9c2e64b53d804601b7311747a3f8dce4 P c184c96b994d4707919e20333d152788: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c2e64b53d804601b7311747a3f8dce4
I20240503 02:21:40.789291 20552 catalog_manager.cc:4830] TS c184c96b994d4707919e20333d152788 (127.20.218.67:41081): tablet 9c2e64b53d804601b7311747a3f8dce4 (table rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]) successfully deleted
I20240503 02:21:40.810572 21689 raft_consensus.cc:1050] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: attempting to promote NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d to VOTER
I20240503 02:21:40.811484 21689 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.814112 21767 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.814105 20837 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.814105 20968 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.814999 22262 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.815450 22180 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.815789 21689 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.823323 21689 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.824155 20968 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.824167 21767 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.828500 20837 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.828969 20556 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.835093 20705 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.836885 21767 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.837092 20837 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.837522 21689 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.837842 22262 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.840160 22179 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.840711 21767 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.844312 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 080b798481e34ca69b0df5141e439567 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:40.845629 20837 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.846798 20564 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:40.850925 20949 tablet_service.cc:1514] Processing DeleteTablet for tablet 080b798481e34ca69b0df5141e439567 with delete_type TABLET_DATA_TOMBSTONED (TS c184c96b994d4707919e20333d152788 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35000
I20240503 02:21:40.851322 22260 tablet_replica.cc:330] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: stopping tablet replica
I20240503 02:21:40.851578 22260 raft_consensus.cc:2229] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:40.851953 22260 raft_consensus.cc:2258] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:40.853017 22260 ts_tablet_manager.cc:1875] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:40.857578 22260 ts_tablet_manager.cc:1888] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:40.857712 22260 log.cc:1196] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/080b798481e34ca69b0df5141e439567
I20240503 02:21:40.858528 20552 catalog_manager.cc:4830] TS c184c96b994d4707919e20333d152788 (127.20.218.67:41081): tablet 080b798481e34ca69b0df5141e439567 (table rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]) successfully deleted
I20240503 02:21:40.901798 22166 raft_consensus.cc:1050] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: attempting to promote NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d to VOTER
I20240503 02:21:40.902679 22166 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.905223 21767 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.905547 20705 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.905608 20837 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.906140 22166 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.906498 22268 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.906915 22266 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.910137 22266 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.911319 20837 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.913542 21767 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.914644 20705 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.915525 20556 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:40.921247 20968 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:40.923096 21767 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.924096 20837 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:40.923684 22268 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:21:40.925168 22269 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:40.927632 22166 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.928351 21767 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.931154 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cde2d2192a041b78adbf768409e2d6d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:40.933058 20565 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d reported cstate change: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.934295 20837 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:40.937258 20686 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cde2d2192a041b78adbf768409e2d6d with delete_type TABLET_DATA_TOMBSTONED (TS 5ba5197bbdb144878029533fd45d53b3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48912
I20240503 02:21:40.937732 22200 tablet_replica.cc:330] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: stopping tablet replica
I20240503 02:21:40.937955 22200 raft_consensus.cc:2229] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:40.938381 22200 raft_consensus.cc:2258] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:40.941807 22200 ts_tablet_manager.cc:1875] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:40.948228 22200 ts_tablet_manager.cc:1888] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:40.948385 22200 log.cc:1196] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/7cde2d2192a041b78adbf768409e2d6d
I20240503 02:21:40.949915 22166 raft_consensus.cc:1050] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788: attempting to promote NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 to VOTER
I20240503 02:21:40.950629 20552 catalog_manager.cc:4830] TS 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): tablet 7cde2d2192a041b78adbf768409e2d6d (table rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]) successfully deleted
I20240503 02:21:40.950747 22166 consensus_queue.cc:237] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:40.957460 20837 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.957460 20705 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.957768 21636 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.958052 22269 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.958547 22268 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.959004 22269 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.966437 21689 raft_consensus.cc:1050] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: attempting to promote NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 to VOTER
I20240503 02:21:40.967635 21689 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:40.968876 22180 raft_consensus.cc:1050] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3: attempting to promote NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 to VOTER
I20240503 02:21:40.969686 22180 consensus_queue.cc:237] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:40.971832 21635 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.972534 22262 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.972862 21635 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.973457 20968 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.974767 20836 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.975416 20838 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.975365 22179 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.976178 22179 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.976923 22179 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.977766 22179 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.979261 20968 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:40.980700 22262 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:40.986382 22269 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.988049 22180 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.990483 21633 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.992561 20705 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.993037 20967 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.997462 20838 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:40.999037 20564 catalog_manager.cc:5484] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:41.005550 20564 catalog_manager.cc:5484] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:41.022887 20838 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.023233 20967 consensus_queue.cc:237] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:41.025429 21633 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.026257 22266 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.028235 20837 raft_consensus.cc:1261] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.030100 22166 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.030830 21633 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.031204 22268 consensus_queue.cc:1035] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.033198 20835 raft_consensus.cc:2932] T a69bdcc01c674a81a3cfd83e9eaaea94 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.034076 22262 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.035995 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a69bdcc01c674a81a3cfd83e9eaaea94 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:41.038966 20556 catalog_manager.cc:5484] T a69bdcc01c674a81a3cfd83e9eaaea94 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 5 to 6, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:41.039299 20967 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.040445 20564 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:41.041118 20835 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.044677 21633 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.046765 21635 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.051950 20705 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:41.053762 20706 consensus_queue.cc:237] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:41.054693 20686 tablet_service.cc:1514] Processing DeleteTablet for tablet a69bdcc01c674a81a3cfd83e9eaaea94 with delete_type TABLET_DATA_TOMBSTONED (TS 5ba5197bbdb144878029533fd45d53b3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48912
I20240503 02:21:41.055100 22200 tablet_replica.cc:330] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: stopping tablet replica
I20240503 02:21:41.055269 22200 raft_consensus.cc:2229] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:41.055593 20835 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.055665 22200 raft_consensus.cc:2258] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:41.056746 22200 ts_tablet_manager.cc:1875] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:41.057349 22180 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.057550 21634 raft_consensus.cc:1261] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.059532 20967 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.060201 21689 consensus_queue.cc:1035] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.061077 22179 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.061663 21689 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.064297 22200 ts_tablet_manager.cc:1888] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:41.064440 22200 log.cc:1196] T a69bdcc01c674a81a3cfd83e9eaaea94 P 5ba5197bbdb144878029533fd45d53b3: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/a69bdcc01c674a81a3cfd83e9eaaea94
I20240503 02:21:41.067010 20556 catalog_manager.cc:5484] T b09ffd8e3f3148a6b1919682c19073a5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:41.067782 20552 catalog_manager.cc:4830] TS 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): tablet a69bdcc01c674a81a3cfd83e9eaaea94 (table rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]) successfully deleted
I20240503 02:21:41.067503 22262 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.068053 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b09ffd8e3f3148a6b1919682c19073a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:41.068167 21634 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.068967 20835 raft_consensus.cc:2932] T b09ffd8e3f3148a6b1919682c19073a5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.072149 20949 tablet_service.cc:1514] Processing DeleteTablet for tablet b09ffd8e3f3148a6b1919682c19073a5 with delete_type TABLET_DATA_TOMBSTONED (TS c184c96b994d4707919e20333d152788 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35000
I20240503 02:21:41.072523 22260 tablet_replica.cc:330] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: stopping tablet replica
I20240503 02:21:41.072723 22260 raft_consensus.cc:2229] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:41.072772 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba1ffdf4cab040a6871acabfc0fec2e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:41.072964 22260 raft_consensus.cc:2258] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:41.073602 20556 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 5 to 6, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:41.074301 22260 ts_tablet_manager.cc:1875] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:41.075902 20967 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.079514 21633 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:21:41.081152 21689 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:41.083413 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1ffdf4cab040a6871acabfc0fec2e5 with delete_type TABLET_DATA_TOMBSTONED (TS 7836d9d893434d7a99d61974b10dd946 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58034
I20240503 02:21:41.083802 21633 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:41.085377 22260 ts_tablet_manager.cc:1888] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:41.085541 22260 log.cc:1196] T b09ffd8e3f3148a6b1919682c19073a5 P c184c96b994d4707919e20333d152788: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/b09ffd8e3f3148a6b1919682c19073a5
I20240503 02:21:41.086449 20552 catalog_manager.cc:4830] TS c184c96b994d4707919e20333d152788 (127.20.218.67:41081): tablet b09ffd8e3f3148a6b1919682c19073a5 (table rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]) successfully deleted
I20240503 02:21:41.087877 22298 tablet_replica.cc:330] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: stopping tablet replica
I20240503 02:21:41.088418 22298 raft_consensus.cc:2229] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:41.088972 22298 raft_consensus.cc:2258] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:41.091197 22298 ts_tablet_manager.cc:1875] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:41.097631 22298 ts_tablet_manager.cc:1888] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:21:41.097915 22298 log.cc:1196] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba1ffdf4cab040a6871acabfc0fec2e5
I20240503 02:21:41.098832 20550 catalog_manager.cc:4830] TS 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): tablet ba1ffdf4cab040a6871acabfc0fec2e5 (table rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]) successfully deleted
I20240503 02:21:41.253172 20686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:41.259758 20949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:41.265594 20818 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:41.269965 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:41.282039 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:41.356003 20835 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:41.358647 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:41.358857 20967 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:41.359599 21960 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:41.360219 21960 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:41.374699 22141 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:41.376216 20706 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:41.376313 20967 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } } }
I20240503 02:21:41.382405 20556 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index -1 to 4, NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) added. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:21:41.383985 20771 consensus_peers.cc:485] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 -> Peer 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045): Couldn't send request to peer 46b0cc47f0274f63a284f07b49cc6da5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44d51a4af792486ab736338c2d377c67. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:41.397835 20835 consensus_queue.cc:237] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:41.400424 20706 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:41.401149 20967 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:21:41.401752 22141 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:21:41.402182 20771 consensus_peers.cc:485] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 -> Peer 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887): Couldn't send request to peer 16c046bf08ee4bd684ff0fcb5b158d4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c431e0d457a44aaafff4abb86415214. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:41.402221 21960 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:21:41.405558 21960 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:41.406379 20706 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:41.411688 20557 catalog_manager.cc:5484] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:41.419073 20967 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } } }
I20240503 02:21:41.945948 22401 ts_tablet_manager.cc:902] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Initiating tablet copy from peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:41.947438 22401 tablet_copy_client.cc:308] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:33349
I20240503 02:21:41.948284 20858 tablet_copy_service.cc:140] P 7836d9d893434d7a99d61974b10dd946: Received BeginTabletCopySession request for tablet 1c431e0d457a44aaafff4abb86415214 from peer 16c046bf08ee4bd684ff0fcb5b158d4d ({username='slave'} at 127.20.218.68:50591)
I20240503 02:21:41.948505 20858 tablet_copy_service.cc:161] P 7836d9d893434d7a99d61974b10dd946: Beginning new tablet copy session on tablet 1c431e0d457a44aaafff4abb86415214 from peer 16c046bf08ee4bd684ff0fcb5b158d4d at {username='slave'} at 127.20.218.68:50591: session id = 16c046bf08ee4bd684ff0fcb5b158d4d-1c431e0d457a44aaafff4abb86415214
I20240503 02:21:41.951613 20858 tablet_copy_source_session.cc:205] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:41.953744 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c431e0d457a44aaafff4abb86415214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:41.962787 22401 tablet_copy_client.cc:783] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:41.963143 22401 tablet_copy_client.cc:647] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:41.964879 22401 tablet_copy_client.cc:523] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:41.967335 22401 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap starting.
I20240503 02:21:41.969830 22406 ts_tablet_manager.cc:902] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Initiating tablet copy from peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349)
I20240503 02:21:41.971940 22406 tablet_copy_client.cc:308] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:33349
I20240503 02:21:41.978972 20858 tablet_copy_service.cc:140] P 7836d9d893434d7a99d61974b10dd946: Received BeginTabletCopySession request for tablet 44d51a4af792486ab736338c2d377c67 from peer 46b0cc47f0274f63a284f07b49cc6da5 ({username='slave'} at 127.20.218.69:45221)
I20240503 02:21:41.979142 20858 tablet_copy_service.cc:161] P 7836d9d893434d7a99d61974b10dd946: Beginning new tablet copy session on tablet 44d51a4af792486ab736338c2d377c67 from peer 46b0cc47f0274f63a284f07b49cc6da5 at {username='slave'} at 127.20.218.69:45221: session id = 46b0cc47f0274f63a284f07b49cc6da5-44d51a4af792486ab736338c2d377c67
I20240503 02:21:41.981096 20858 tablet_copy_source_session.cc:205] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:41.982436 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d51a4af792486ab736338c2d377c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:41.987007 22406 tablet_copy_client.cc:783] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:41.987342 22406 tablet_copy_client.cc:647] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:41.989228 22406 tablet_copy_client.cc:523] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:41.991657 22406 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap starting.
I20240503 02:21:42.002243 22401 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:42.002655 22401 tablet_bootstrap.cc:492] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Bootstrap complete.
I20240503 02:21:42.002910 22401 ts_tablet_manager.cc:1367] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.004s
I20240503 02:21:42.003571 22401 raft_consensus.cc:348] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:42.003791 22401 raft_consensus.cc:729] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c046bf08ee4bd684ff0fcb5b158d4d, State: Initialized, Role: LEARNER
I20240503 02:21:42.004002 22401 consensus_queue.cc:260] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: true } }
I20240503 02:21:42.004756 22401 ts_tablet_manager.cc:1398] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240503 02:21:42.005687 20858 tablet_copy_service.cc:342] P 7836d9d893434d7a99d61974b10dd946: Request end of tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-1c431e0d457a44aaafff4abb86415214 received from {username='slave'} at 127.20.218.68:50591
I20240503 02:21:42.005867 20858 tablet_copy_service.cc:433] P 7836d9d893434d7a99d61974b10dd946: ending tablet copy session 16c046bf08ee4bd684ff0fcb5b158d4d-1c431e0d457a44aaafff4abb86415214 on tablet 1c431e0d457a44aaafff4abb86415214 with peer 16c046bf08ee4bd684ff0fcb5b158d4d
I20240503 02:21:42.024574 22406 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:42.024991 22406 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Bootstrap complete.
I20240503 02:21:42.025252 22406 ts_tablet_manager.cc:1367] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent bootstrapping tablet: real 0.034s	user 0.032s	sys 0.000s
I20240503 02:21:42.025983 22406 raft_consensus.cc:348] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:42.026197 22406 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46b0cc47f0274f63a284f07b49cc6da5, State: Initialized, Role: LEARNER
I20240503 02:21:42.026393 22406 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: NON_VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: true } }
I20240503 02:21:42.027128 22406 ts_tablet_manager.cc:1398] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:42.027997 20858 tablet_copy_service.cc:342] P 7836d9d893434d7a99d61974b10dd946: Request end of tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-44d51a4af792486ab736338c2d377c67 received from {username='slave'} at 127.20.218.69:45221
I20240503 02:21:42.028143 20858 tablet_copy_service.cc:433] P 7836d9d893434d7a99d61974b10dd946: ending tablet copy session 46b0cc47f0274f63a284f07b49cc6da5-44d51a4af792486ab736338c2d377c67 on tablet 44d51a4af792486ab736338c2d377c67 with peer 46b0cc47f0274f63a284f07b49cc6da5
I20240503 02:21:42.504114 21767 raft_consensus.cc:1203] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:42.569965 21633 raft_consensus.cc:1203] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:21:43.083967 21960 raft_consensus.cc:1050] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: attempting to promote NON_VOTER 16c046bf08ee4bd684ff0fcb5b158d4d to VOTER
I20240503 02:21:43.084669 21960 consensus_queue.cc:237] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:43.086938 21767 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 LEARNER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.087132 20967 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.087529 22471 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.087656 20706 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.087965 22469 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.088275 21960 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.091586 21960 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:43.092350 20967 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:43.092397 20706 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:43.095324 21767 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:43.099148 20557 catalog_manager.cc:5484] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 4 to 5, 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:43.132587 22471 raft_consensus.cc:1050] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: attempting to promote NON_VOTER 46b0cc47f0274f63a284f07b49cc6da5 to VOTER
I20240503 02:21:43.133401 22471 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:43.136056 20967 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.136061 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 LEARNER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.136116 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer 7836d9d893434d7a99d61974b10dd946: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:21:43.136656 22471 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.137188 22481 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.137566 22469 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:21:43.146160 22471 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:43.146994 20706 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:43.147433 21633 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:43.150691 20967 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:43.151296 20556 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 4 to 5, 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7836d9d893434d7a99d61974b10dd946" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:43.199476 20835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214"
dest_uuid: "7836d9d893434d7a99d61974b10dd946"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46792
I20240503 02:21:43.199760 20835 raft_consensus.cc:595] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:43.217597 20835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d51a4af792486ab736338c2d377c67"
dest_uuid: "7836d9d893434d7a99d61974b10dd946"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46806
I20240503 02:21:43.217808 20835 raft_consensus.cc:595] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:43.436003 20835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214"
dest_uuid: "7836d9d893434d7a99d61974b10dd946"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46828
I20240503 02:21:43.436221 20835 raft_consensus.cc:595] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:43.455363 20835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d51a4af792486ab736338c2d377c67"
dest_uuid: "7836d9d893434d7a99d61974b10dd946"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46840
I20240503 02:21:43.455565 20835 raft_consensus.cc:595] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:43.482331 22481 raft_consensus.cc:979] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: : Instructing follower 5ba5197bbdb144878029533fd45d53b3 to start an election
I20240503 02:21:43.482532 22481 raft_consensus.cc:1067] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Signalling peer 5ba5197bbdb144878029533fd45d53b3 to start an election
I20240503 02:21:43.483506 20706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214"
dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
 from {username='slave'} at 127.20.218.66:36111
I20240503 02:21:43.483770 20706 raft_consensus.cc:482] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:21:43.483889 20706 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.486121 20706 raft_consensus.cc:504] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:43.487339 20835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:43.487428 20967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c184c96b994d4707919e20333d152788"
I20240503 02:21:43.487607 20835 raft_consensus.cc:3032] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:21:43.487665 20967 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.487725 20835 raft_consensus.cc:729] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:43.487650 21767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c431e0d457a44aaafff4abb86415214" candidate_uuid: "5ba5197bbdb144878029533fd45d53b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d"
I20240503 02:21:43.487805 20706 leader_election.cc:288] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 2 election: Requested vote from peers 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), c184c96b994d4707919e20333d152788 (127.20.218.67:41081), 16c046bf08ee4bd684ff0fcb5b158d4d (127.20.218.68:36887)
I20240503 02:21:43.488026 21767 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.488068 20835 consensus_queue.cc:260] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:43.488626 20835 raft_consensus.cc:3037] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.490438 21767 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 2.
I20240503 02:21:43.490736 20967 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 2.
I20240503 02:21:43.491185 20642 leader_election.cc:302] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 16c046bf08ee4bd684ff0fcb5b158d4d, 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:43.491395 20835 raft_consensus.cc:2454] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba5197bbdb144878029533fd45d53b3 in term 2.
I20240503 02:21:43.491516 22486 raft_consensus.cc:2781] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:21:43.491667 22486 raft_consensus.cc:686] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 2 LEADER]: Becoming Leader. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Running, Role: LEADER
I20240503 02:21:43.492060 22486 consensus_queue.cc:237] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:43.494689 20565 catalog_manager.cc:5484] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: term changed from 1 to 2, leader changed from 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) to 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65). New cstate: current_term: 2 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:43.686443 20835 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d51a4af792486ab736338c2d377c67"
dest_uuid: "7836d9d893434d7a99d61974b10dd946"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46846
I20240503 02:21:43.686642 20835 raft_consensus.cc:595] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:21:43.743053 22471 raft_consensus.cc:979] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: : Instructing follower c184c96b994d4707919e20333d152788 to start an election
I20240503 02:21:43.743260 22471 raft_consensus.cc:1067] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Signalling peer c184c96b994d4707919e20333d152788 to start an election
I20240503 02:21:43.744110 20967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44d51a4af792486ab736338c2d377c67"
dest_uuid: "c184c96b994d4707919e20333d152788"
 from {username='slave'} at 127.20.218.66:46661
I20240503 02:21:43.744359 20967 raft_consensus.cc:482] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:21:43.744468 20967 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.746587 20967 raft_consensus.cc:504] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:43.747313 20967 leader_election.cc:288] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 2 election: Requested vote from peers 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579), 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349), 46b0cc47f0274f63a284f07b49cc6da5 (127.20.218.69:44045)
I20240503 02:21:43.747867 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5ba5197bbdb144878029533fd45d53b3"
I20240503 02:21:43.748082 20706 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.748064 20835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7836d9d893434d7a99d61974b10dd946"
I20240503 02:21:43.748082 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d51a4af792486ab736338c2d377c67" candidate_uuid: "c184c96b994d4707919e20333d152788" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "46b0cc47f0274f63a284f07b49cc6da5"
I20240503 02:21:43.748281 20835 raft_consensus.cc:3032] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:21:43.748385 20835 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Running, Role: LEADER
I20240503 02:21:43.748432 21633 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.748721 20835 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:43.749207 20835 raft_consensus.cc:3037] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:21:43.750113 20706 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 2.
I20240503 02:21:43.751374 21633 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 2.
I20240503 02:21:43.751968 20902 leader_election.cc:302] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 46b0cc47f0274f63a284f07b49cc6da5, 5ba5197bbdb144878029533fd45d53b3, c184c96b994d4707919e20333d152788; no voters: 
I20240503 02:21:43.752038 20835 raft_consensus.cc:2454] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c184c96b994d4707919e20333d152788 in term 2.
I20240503 02:21:43.752287 22266 raft_consensus.cc:2781] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:21:43.752799 22266 raft_consensus.cc:686] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 LEADER]: Becoming Leader. State: Replica: c184c96b994d4707919e20333d152788, State: Running, Role: LEADER
I20240503 02:21:43.753144 22266 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:43.756762 20565 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 reported cstate change: term changed from 1 to 2, leader changed from 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) to c184c96b994d4707919e20333d152788 (127.20.218.67). New cstate: current_term: 2 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:43.913740 21767 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 2 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:43.914327 22486 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:43.997792 20835 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:43.998456 22486 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:44.003219 20967 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:44.004122 22486 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:44.159013 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:44.159655 22266 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:44.223987 20836 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:44.224622 22266 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:44.228974 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:21:44.229992 22574 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:21:44.514137 20706 consensus_queue.cc:237] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } }
I20240503 02:21:44.515956 21767 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 2 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:21:44.515949 20967 raft_consensus.cc:1261] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:21:44.516646 22558 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:44.517035 22556 consensus_queue.cc:1035] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:44.519452 22486 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 5ba5197bbdb144878029533fd45d53b3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:44.520201 20967 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:44.522125 21767 raft_consensus.cc:2932] T 1c431e0d457a44aaafff4abb86415214 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:44.522974 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c431e0d457a44aaafff4abb86415214 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:44.525012 20564 catalog_manager.cc:5484] T 1c431e0d457a44aaafff4abb86415214 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } }
I20240503 02:21:44.529491 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c431e0d457a44aaafff4abb86415214 with delete_type TABLET_DATA_TOMBSTONED (TS 7836d9d893434d7a99d61974b10dd946 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58034
I20240503 02:21:44.530113 22592 tablet_replica.cc:330] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: stopping tablet replica
I20240503 02:21:44.530349 22592 raft_consensus.cc:2229] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:44.530611 22592 raft_consensus.cc:2258] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:44.531441 22592 ts_tablet_manager.cc:1875] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:44.535874 22592 ts_tablet_manager.cc:1888] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:21:44.536026 22592 log.cc:1196] T 1c431e0d457a44aaafff4abb86415214 P 7836d9d893434d7a99d61974b10dd946: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c431e0d457a44aaafff4abb86415214
I20240503 02:21:44.536950 20550 catalog_manager.cc:4830] TS 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): tablet 1c431e0d457a44aaafff4abb86415214 (table rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]) successfully deleted
I20240503 02:21:44.859846 20967 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } }
I20240503 02:21:44.861893 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:21:44.861949 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:21:44.862506 22615 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:44.862833 22266 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:44.865264 22613 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:44.865919 20706 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:44.866133 21633 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:44.869236 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44d51a4af792486ab736338c2d377c67 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:44.870923 20565 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 5 to 7, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New cstate: current_term: 2 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } }
I20240503 02:21:44.876685 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d51a4af792486ab736338c2d377c67 with delete_type TABLET_DATA_TOMBSTONED (TS 7836d9d893434d7a99d61974b10dd946 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58034
I20240503 02:21:44.877243 22592 tablet_replica.cc:330] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: stopping tablet replica
I20240503 02:21:44.877563 22592 raft_consensus.cc:2229] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:44.877830 22592 raft_consensus.cc:2258] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:44.879118 22592 ts_tablet_manager.cc:1875] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:44.884639 22592 ts_tablet_manager.cc:1888] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:21:44.884763 22592 log.cc:1196] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/44d51a4af792486ab736338c2d377c67
I20240503 02:21:44.885553 20550 catalog_manager.cc:4830] TS 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): tablet 44d51a4af792486ab736338c2d377c67 (table rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]) successfully deleted
I20240503 02:21:45.107093 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:45.118561 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:45.132046 20686 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:45.147576 20818 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:45.161186 20949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:45.325522 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:45.325723 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:45.333460 20686 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:45.342271 20818 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:45.346913 20949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:45.422138 20967 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.424257 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:21:45.424425 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:21:45.424944 22698 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:45.425288 22615 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:21:45.425679 20903 consensus_peers.cc:485] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 -> Peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): Couldn't send request to peer 7836d9d893434d7a99d61974b10dd946. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:45.427690 22266 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.428333 20706 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.430042 21633 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.432777 20564 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 7 to 8, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New cstate: current_term: 2 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:45.448361 20706 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.450956 21633 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:45.451213 20967 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:21:45.451401 20640 consensus_peers.cc:485] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 -> Peer 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): Couldn't send request to peer 7836d9d893434d7a99d61974b10dd946. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:45.451748 22709 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:45.452078 22556 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:45.455026 22709 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.455874 21633 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.456313 20967 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.461292 20565 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 reported cstate change: config changed from index 6 to 7, NON_VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } } }
I20240503 02:21:45.474660 20967 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } }
I20240503 02:21:45.477227 20836 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:45.477263 21767 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:45.477797 22615 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:45.478106 22698 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:21:45.478390 20905 consensus_peers.cc:485] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 -> Peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): Couldn't send request to peer 5ba5197bbdb144878029533fd45d53b3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:45.480684 22266 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } } }
I20240503 02:21:45.481374 20836 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } } }
I20240503 02:21:45.481379 21767 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } } }
I20240503 02:21:45.486061 20564 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 6 to 7, NON_VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) added. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:45.500828 20706 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } }
W20240503 02:21:45.503384 20642 consensus_peers.cc:485] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 -> Peer c184c96b994d4707919e20333d152788 (127.20.218.67:41081): Couldn't send request to peer c184c96b994d4707919e20333d152788. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:45.503510 20836 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:45.503693 21767 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:21:45.504108 22709 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:45.504431 22727 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:21:45.507427 22709 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } } }
I20240503 02:21:45.508359 20836 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } } }
I20240503 02:21:45.508664 21767 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } } }
I20240503 02:21:45.513048 20556 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 6 to 7, NON_VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) added. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:45.812721 22770 ts_tablet_manager.cc:902] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Initiating tablet copy from peer c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:45.814371 22770 tablet_copy_client.cc:308] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41081
I20240503 02:21:45.815261 20989 tablet_copy_service.cc:140] P c184c96b994d4707919e20333d152788: Received BeginTabletCopySession request for tablet 44d51a4af792486ab736338c2d377c67 from peer 7836d9d893434d7a99d61974b10dd946 ({username='slave'} at 127.20.218.66:46661)
I20240503 02:21:45.815425 20989 tablet_copy_service.cc:161] P c184c96b994d4707919e20333d152788: Beginning new tablet copy session on tablet 44d51a4af792486ab736338c2d377c67 from peer 7836d9d893434d7a99d61974b10dd946 at {username='slave'} at 127.20.218.66:46661: session id = 7836d9d893434d7a99d61974b10dd946-44d51a4af792486ab736338c2d377c67
I20240503 02:21:45.817530 20989 tablet_copy_source_session.cc:205] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:45.818863 22770 ts_tablet_manager.cc:1875] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:45.824236 22770 ts_tablet_manager.cc:1888] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 02:21:45.824535 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d51a4af792486ab736338c2d377c67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:45.827296 22770 tablet_copy_client.cc:783] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:45.827625 22770 tablet_copy_client.cc:647] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:45.829780 22770 tablet_copy_client.cc:523] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:45.832777 22770 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:45.867272 22770 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:45.867774 22770 tablet_bootstrap.cc:492] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Bootstrap complete.
I20240503 02:21:45.868065 22770 ts_tablet_manager.cc:1367] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.035s	user 0.032s	sys 0.004s
I20240503 02:21:45.868767 22770 raft_consensus.cc:348] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.869014 22770 raft_consensus.cc:729] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: LEARNER
I20240503 02:21:45.869200 22770 consensus_queue.cc:260] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.869884 22770 ts_tablet_manager.cc:1398] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:21:45.870810 20989 tablet_copy_service.cc:342] P c184c96b994d4707919e20333d152788: Request end of tablet copy session 7836d9d893434d7a99d61974b10dd946-44d51a4af792486ab736338c2d377c67 received from {username='slave'} at 127.20.218.66:46661
I20240503 02:21:45.870965 20989 tablet_copy_service.cc:433] P c184c96b994d4707919e20333d152788: ending tablet copy session 7836d9d893434d7a99d61974b10dd946-44d51a4af792486ab736338c2d377c67 on tablet 44d51a4af792486ab736338c2d377c67 with peer 7836d9d893434d7a99d61974b10dd946
I20240503 02:21:45.925020 22770 ts_tablet_manager.cc:902] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Initiating tablet copy from peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:45.926359 22770 tablet_copy_client.cc:308] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:43579
I20240503 02:21:45.927120 20726 tablet_copy_service.cc:140] P 5ba5197bbdb144878029533fd45d53b3: Received BeginTabletCopySession request for tablet ba1ffdf4cab040a6871acabfc0fec2e5 from peer 7836d9d893434d7a99d61974b10dd946 ({username='slave'} at 127.20.218.66:36111)
I20240503 02:21:45.927302 20726 tablet_copy_service.cc:161] P 5ba5197bbdb144878029533fd45d53b3: Beginning new tablet copy session on tablet ba1ffdf4cab040a6871acabfc0fec2e5 from peer 7836d9d893434d7a99d61974b10dd946 at {username='slave'} at 127.20.218.66:36111: session id = 7836d9d893434d7a99d61974b10dd946-ba1ffdf4cab040a6871acabfc0fec2e5
I20240503 02:21:45.929265 20726 tablet_copy_source_session.cc:205] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:45.930298 22770 ts_tablet_manager.cc:1875] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:45.934773 22770 ts_tablet_manager.cc:1888] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:21:45.935032 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1ffdf4cab040a6871acabfc0fec2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:45.937140 22770 tablet_copy_client.cc:783] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:45.937388 22770 tablet_copy_client.cc:647] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:45.939033 22770 tablet_copy_client.cc:523] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:45.941407 22770 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Bootstrap starting.
I20240503 02:21:45.957427 22781 ts_tablet_manager.cc:902] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Initiating tablet copy from peer c184c96b994d4707919e20333d152788 (127.20.218.67:41081)
I20240503 02:21:45.959255 22781 tablet_copy_client.cc:308] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41081
I20240503 02:21:45.960379 20989 tablet_copy_service.cc:140] P c184c96b994d4707919e20333d152788: Received BeginTabletCopySession request for tablet 7cde2d2192a041b78adbf768409e2d6d from peer 5ba5197bbdb144878029533fd45d53b3 ({username='slave'} at 127.20.218.65:51405)
I20240503 02:21:45.960587 20989 tablet_copy_service.cc:161] P c184c96b994d4707919e20333d152788: Beginning new tablet copy session on tablet 7cde2d2192a041b78adbf768409e2d6d from peer 5ba5197bbdb144878029533fd45d53b3 at {username='slave'} at 127.20.218.65:51405: session id = 5ba5197bbdb144878029533fd45d53b3-7cde2d2192a041b78adbf768409e2d6d
I20240503 02:21:45.963569 20989 tablet_copy_source_session.cc:205] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:45.965087 22781 ts_tablet_manager.cc:1875] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:45.970427 22781 ts_tablet_manager.cc:1888] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:21:45.970731 22781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cde2d2192a041b78adbf768409e2d6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:45.973742 22781 tablet_copy_client.cc:783] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:45.974071 22781 tablet_copy_client.cc:647] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:45.976038 22781 tablet_copy_client.cc:523] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:45.978794 22770 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:45.979207 22781 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Bootstrap starting.
I20240503 02:21:45.979218 22770 tablet_bootstrap.cc:492] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Bootstrap complete.
I20240503 02:21:45.979460 22770 ts_tablet_manager.cc:1367] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Time spent bootstrapping tablet: real 0.038s	user 0.037s	sys 0.000s
I20240503 02:21:45.980126 22770 raft_consensus.cc:348] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.980340 22770 raft_consensus.cc:729] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7836d9d893434d7a99d61974b10dd946, State: Initialized, Role: LEARNER
I20240503 02:21:45.980561 22770 consensus_queue.cc:260] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: true } }
I20240503 02:21:45.981055 22770 ts_tablet_manager.cc:1398] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:21:45.981959 20726 tablet_copy_service.cc:342] P 5ba5197bbdb144878029533fd45d53b3: Request end of tablet copy session 7836d9d893434d7a99d61974b10dd946-ba1ffdf4cab040a6871acabfc0fec2e5 received from {username='slave'} at 127.20.218.66:36111
I20240503 02:21:45.982100 20726 tablet_copy_service.cc:433] P 5ba5197bbdb144878029533fd45d53b3: ending tablet copy session 7836d9d893434d7a99d61974b10dd946-ba1ffdf4cab040a6871acabfc0fec2e5 on tablet ba1ffdf4cab040a6871acabfc0fec2e5 with peer 7836d9d893434d7a99d61974b10dd946
I20240503 02:21:46.026260 22803 ts_tablet_manager.cc:902] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Initiating tablet copy from peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579)
I20240503 02:21:46.028746 22781 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:46.029247 22781 tablet_bootstrap.cc:492] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Bootstrap complete.
I20240503 02:21:46.029554 22781 ts_tablet_manager.cc:1367] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Time spent bootstrapping tablet: real 0.050s	user 0.047s	sys 0.004s
I20240503 02:21:46.030269 22781 raft_consensus.cc:348] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } }
I20240503 02:21:46.030486 22781 raft_consensus.cc:729] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ba5197bbdb144878029533fd45d53b3, State: Initialized, Role: LEARNER
I20240503 02:21:46.030721 22781 consensus_queue.cc:260] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: true } }
I20240503 02:21:46.031405 22781 ts_tablet_manager.cc:1398] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:46.032380 20989 tablet_copy_service.cc:342] P c184c96b994d4707919e20333d152788: Request end of tablet copy session 5ba5197bbdb144878029533fd45d53b3-7cde2d2192a041b78adbf768409e2d6d received from {username='slave'} at 127.20.218.65:51405
I20240503 02:21:46.032516 20989 tablet_copy_service.cc:433] P c184c96b994d4707919e20333d152788: ending tablet copy session 5ba5197bbdb144878029533fd45d53b3-7cde2d2192a041b78adbf768409e2d6d on tablet 7cde2d2192a041b78adbf768409e2d6d with peer 5ba5197bbdb144878029533fd45d53b3
I20240503 02:21:46.037262 22803 tablet_copy_client.cc:308] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:43579
I20240503 02:21:46.038257 20726 tablet_copy_service.cc:140] P 5ba5197bbdb144878029533fd45d53b3: Received BeginTabletCopySession request for tablet 080b798481e34ca69b0df5141e439567 from peer c184c96b994d4707919e20333d152788 ({username='slave'} at 127.20.218.67:34361)
I20240503 02:21:46.038424 20726 tablet_copy_service.cc:161] P 5ba5197bbdb144878029533fd45d53b3: Beginning new tablet copy session on tablet 080b798481e34ca69b0df5141e439567 from peer c184c96b994d4707919e20333d152788 at {username='slave'} at 127.20.218.67:34361: session id = c184c96b994d4707919e20333d152788-080b798481e34ca69b0df5141e439567
I20240503 02:21:46.040959 20726 tablet_copy_source_session.cc:205] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:21:46.042589 22803 ts_tablet_manager.cc:1875] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:21:46.047996 22803 ts_tablet_manager.cc:1888] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:21:46.048310 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 080b798481e34ca69b0df5141e439567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:46.050997 22803 tablet_copy_client.cc:783] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet copy: Starting download of 0 data blocks...
I20240503 02:21:46.051294 22803 tablet_copy_client.cc:647] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet copy: Starting download of 1 WAL segments...
I20240503 02:21:46.053296 22803 tablet_copy_client.cc:523] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:21:46.056547 22803 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Bootstrap starting.
I20240503 02:21:46.091413 22803 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:21:46.091894 22803 tablet_bootstrap.cc:492] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Bootstrap complete.
I20240503 02:21:46.092177 22803 ts_tablet_manager.cc:1367] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.004s
I20240503 02:21:46.092856 22803 raft_consensus.cc:348] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } }
I20240503 02:21:46.093086 22803 raft_consensus.cc:729] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c184c96b994d4707919e20333d152788, State: Initialized, Role: LEARNER
I20240503 02:21:46.093292 22803 consensus_queue.cc:260] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: NON_VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: true } }
I20240503 02:21:46.093897 22803 ts_tablet_manager.cc:1398] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:21:46.094827 20726 tablet_copy_service.cc:342] P 5ba5197bbdb144878029533fd45d53b3: Request end of tablet copy session c184c96b994d4707919e20333d152788-080b798481e34ca69b0df5141e439567 received from {username='slave'} at 127.20.218.67:34361
I20240503 02:21:46.094956 20726 tablet_copy_service.cc:433] P 5ba5197bbdb144878029533fd45d53b3: ending tablet copy session c184c96b994d4707919e20333d152788-080b798481e34ca69b0df5141e439567 on tablet 080b798481e34ca69b0df5141e439567 with peer c184c96b994d4707919e20333d152788
I20240503 02:21:46.290581 20836 raft_consensus.cc:1203] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240503 02:21:46.379217 20706 raft_consensus.cc:1203] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:21:46.410079 20836 raft_consensus.cc:1203] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:21:46.590771 20967 raft_consensus.cc:1203] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240503 02:21:46.820477 22837 raft_consensus.cc:1050] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3: attempting to promote NON_VOTER 7836d9d893434d7a99d61974b10dd946 to VOTER
I20240503 02:21:46.821224 22837 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } }
I20240503 02:21:46.823570 21633 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.823650 20967 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.823820 20836 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 LEARNER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.824304 22709 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.824712 22879 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.825047 22837 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.828130 22879 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.829015 20836 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.829257 20967 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.833854 20557 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 7 to 8, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.834074 21633 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.841125 20706 consensus_queue.cc:237] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } }
I20240503 02:21:46.843005 21633 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:46.843180 20836 raft_consensus.cc:1261] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:46.843597 22837 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.843957 22836 consensus_queue.cc:1035] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.846546 22709 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.847167 21633 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 46b0cc47f0274f63a284f07b49cc6da5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.847247 20836 raft_consensus.cc:2932] T ba1ffdf4cab040a6871acabfc0fec2e5 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.850445 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba1ffdf4cab040a6871acabfc0fec2e5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:46.852232 20557 catalog_manager.cc:5484] T ba1ffdf4cab040a6871acabfc0fec2e5 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 8 to 9, VOTER c184c96b994d4707919e20333d152788 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:46.856405 20949 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1ffdf4cab040a6871acabfc0fec2e5 with delete_type TABLET_DATA_TOMBSTONED (TS c184c96b994d4707919e20333d152788 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35000
I20240503 02:21:46.857124 22888 tablet_replica.cc:330] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: stopping tablet replica
I20240503 02:21:46.857309 22888 raft_consensus.cc:2229] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:46.857910 22888 raft_consensus.cc:2258] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:46.858896 22888 ts_tablet_manager.cc:1875] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:46.864040 22888 ts_tablet_manager.cc:1888] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:21:46.864161 22888 log.cc:1196] T ba1ffdf4cab040a6871acabfc0fec2e5 P c184c96b994d4707919e20333d152788: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba1ffdf4cab040a6871acabfc0fec2e5
I20240503 02:21:46.864923 20552 catalog_manager.cc:4830] TS c184c96b994d4707919e20333d152788 (127.20.218.67:41081): tablet ba1ffdf4cab040a6871acabfc0fec2e5 (table rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]) successfully deleted
I20240503 02:21:46.867547 22835 raft_consensus.cc:1050] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788: attempting to promote NON_VOTER 7836d9d893434d7a99d61974b10dd946 to VOTER
I20240503 02:21:46.868240 22835 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } }
I20240503 02:21:46.870744 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:21:46.870745 20706 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:21:46.870813 20836 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 LEARNER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:21:46.871402 22266 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.871754 22834 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.872103 22835 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.875412 22835 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.876066 20836 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.876643 21633 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.879993 20706 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.880627 20556 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 8 to 9, 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.885905 20967 consensus_queue.cc:237] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } }
I20240503 02:21:46.887553 20836 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:46.887728 21633 raft_consensus.cc:1261] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:21:46.888275 22890 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:46.888837 22890 consensus_queue.cc:1035] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:21:46.890690 22890 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P c184c96b994d4707919e20333d152788 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.891534 21633 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 46b0cc47f0274f63a284f07b49cc6da5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.891405 20836 raft_consensus.cc:2932] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.895972 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44d51a4af792486ab736338c2d377c67 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:46.896744 20564 catalog_manager.cc:5484] T 44d51a4af792486ab736338c2d377c67 P 7836d9d893434d7a99d61974b10dd946 reported cstate change: config changed from index 9 to 10, VOTER 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "46b0cc47f0274f63a284f07b49cc6da5" member_type: VOTER last_known_addr { host: "127.20.218.69" port: 44045 } attrs { promote: false } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { promote: false } } }
I20240503 02:21:46.900363 20686 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d51a4af792486ab736338c2d377c67 with delete_type TABLET_DATA_TOMBSTONED (TS 5ba5197bbdb144878029533fd45d53b3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48912
I20240503 02:21:46.901818 22894 tablet_replica.cc:330] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: stopping tablet replica
I20240503 02:21:46.902024 22894 raft_consensus.cc:2229] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:46.902536 22894 raft_consensus.cc:2258] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:46.903566 22894 ts_tablet_manager.cc:1875] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:46.907974 22894 ts_tablet_manager.cc:1888] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:21:46.908084 22894 log.cc:1196] T 44d51a4af792486ab736338c2d377c67 P 5ba5197bbdb144878029533fd45d53b3: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/44d51a4af792486ab736338c2d377c67
I20240503 02:21:46.908778 20552 catalog_manager.cc:4830] TS 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): tablet 44d51a4af792486ab736338c2d377c67 (table rebalance_test_table_2 [id=22ced3bc7caf489b869326db043bb0ab]) successfully deleted
I20240503 02:21:46.934589 22835 raft_consensus.cc:1050] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788: attempting to promote NON_VOTER 5ba5197bbdb144878029533fd45d53b3 to VOTER
I20240503 02:21:46.935262 22835 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } }
I20240503 02:21:46.937410 20706 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEARNER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.937494 21767 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.937693 20836 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:46.938097 22890 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.938452 22835 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.938738 22266 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:46.942546 22890 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.943292 21767 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.947671 20706 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.948351 20556 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 7 to 8, 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:46.953305 20967 consensus_queue.cc:237] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } }
I20240503 02:21:46.947065 20836 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.957181 21767 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:46.958207 22266 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.959461 20705 raft_consensus.cc:1261] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Refusing update from remote peer c184c96b994d4707919e20333d152788: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:46.960579 22266 consensus_queue.cc:1035] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:46.963006 22835 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.964057 20705 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 5ba5197bbdb144878029533fd45d53b3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.967235 21767 raft_consensus.cc:2932] T 7cde2d2192a041b78adbf768409e2d6d P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } } }
I20240503 02:21:46.967798 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cde2d2192a041b78adbf768409e2d6d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:46.968744 20556 catalog_manager.cc:5484] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "c184c96b994d4707919e20333d152788" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:46.973459 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cde2d2192a041b78adbf768409e2d6d with delete_type TABLET_DATA_TOMBSTONED (TS 7836d9d893434d7a99d61974b10dd946 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58034
I20240503 02:21:46.974216 22904 tablet_replica.cc:330] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: stopping tablet replica
I20240503 02:21:46.974648 22904 raft_consensus.cc:2229] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:46.975032 22904 raft_consensus.cc:2258] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:46.976372 22904 ts_tablet_manager.cc:1875] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:46.981830 22904 ts_tablet_manager.cc:1888] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:21:46.982448 22904 log.cc:1196] T 7cde2d2192a041b78adbf768409e2d6d P 7836d9d893434d7a99d61974b10dd946: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/7cde2d2192a041b78adbf768409e2d6d
I20240503 02:21:46.983345 20550 catalog_manager.cc:4830] TS 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): tablet 7cde2d2192a041b78adbf768409e2d6d (table rebalance_test_table_1 [id=1c38c43e3706426aac1c70955a41f551]) successfully deleted
I20240503 02:21:47.079084 22837 raft_consensus.cc:1050] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3: attempting to promote NON_VOTER c184c96b994d4707919e20333d152788 to VOTER
I20240503 02:21:47.079780 22837 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } }
I20240503 02:21:47.082010 21767 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:47.082136 20967 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 LEARNER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:47.082157 20836 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:21:47.082790 22709 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:47.083103 22837 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:47.083367 22836 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:21:47.090615 22709 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c184c96b994d4707919e20333d152788 (127.20.218.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.091410 21767 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c184c96b994d4707919e20333d152788 (127.20.218.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.091686 20967 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c184c96b994d4707919e20333d152788 (127.20.218.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.095687 20836 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c184c96b994d4707919e20333d152788 (127.20.218.67) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.096235 20557 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 reported cstate change: config changed from index 7 to 8, c184c96b994d4707919e20333d152788 (127.20.218.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "7836d9d893434d7a99d61974b10dd946" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 33349 } attrs { replace: true } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.102147 20705 consensus_queue.cc:237] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } }
I20240503 02:21:47.104085 20967 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:47.104298 21767 raft_consensus.cc:1261] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Refusing update from remote peer 5ba5197bbdb144878029533fd45d53b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:21:47.104729 22836 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:47.105307 22836 consensus_queue.cc:1035] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:21:47.106985 22879 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.107605 20967 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P c184c96b994d4707919e20333d152788 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.108240 21767 raft_consensus.cc:2932] T 080b798481e34ca69b0df5141e439567 P 16c046bf08ee4bd684ff0fcb5b158d4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } } }
I20240503 02:21:47.110458 20552 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 080b798481e34ca69b0df5141e439567 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:21:47.111665 20564 catalog_manager.cc:5484] T 080b798481e34ca69b0df5141e439567 P 5ba5197bbdb144878029533fd45d53b3 reported cstate change: config changed from index 8 to 9, VOTER 7836d9d893434d7a99d61974b10dd946 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "5ba5197bbdb144878029533fd45d53b3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5ba5197bbdb144878029533fd45d53b3" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c046bf08ee4bd684ff0fcb5b158d4d" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 36887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c184c96b994d4707919e20333d152788" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:21:47.115558 20818 tablet_service.cc:1514] Processing DeleteTablet for tablet 080b798481e34ca69b0df5141e439567 with delete_type TABLET_DATA_TOMBSTONED (TS 7836d9d893434d7a99d61974b10dd946 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58034
I20240503 02:21:47.115917 22904 tablet_replica.cc:330] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: stopping tablet replica
I20240503 02:21:47.116124 22904 raft_consensus.cc:2229] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:21:47.116621 22904 raft_consensus.cc:2258] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:21:47.117555 22904 ts_tablet_manager.cc:1875] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:21:47.121845 22904 ts_tablet_manager.cc:1888] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:21:47.121969 22904 log.cc:1196] T 080b798481e34ca69b0df5141e439567 P 7836d9d893434d7a99d61974b10dd946: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/080b798481e34ca69b0df5141e439567
I20240503 02:21:47.122668 20550 catalog_manager.cc:4830] TS 7836d9d893434d7a99d61974b10dd946 (127.20.218.66:33349): tablet 080b798481e34ca69b0df5141e439567 (table rebalance_test_table_0 [id=c7f36142ec21420dbe33ddace469e1bd]) successfully deleted
I20240503 02:21:47.345660 20686 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:47.345719 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:47.353914 20949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:47.355963 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:47.369558 20818 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:47.525589 21748 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:47.528177 21616 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:21:47.538192 20949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:21:47.542321 20818 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:21:47.544200 20686 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:21:47.767045 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20624
I20240503 02:21:47.787442 20746 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:47.935711 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20624
W20240503 02:21:47.971935 21571 connection.cc:535] server connection from 127.20.218.65:40331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:21:47.972658 20772 connection.cc:535] server connection from 127.20.218.65:45791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:47.973565 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20755
I20240503 02:21:47.979697 20878 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:21:48.072825 20905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.218.65:43579: connect: Connection refused (error 111)
W20240503 02:21:48.074339 20905 consensus_peers.cc:485] T 7cde2d2192a041b78adbf768409e2d6d P c184c96b994d4707919e20333d152788 -> Peer 5ba5197bbdb144878029533fd45d53b3 (127.20.218.65:43579): Couldn't send request to peer 5ba5197bbdb144878029533fd45d53b3. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:43579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:21:48.128044 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20755
W20240503 02:21:48.156528 20552 connection.cc:535] server connection from 127.20.218.66:44711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:21:48.156783 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20887
I20240503 02:21:48.163138 21009 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:48.307929 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20887
I20240503 02:21:48.334775 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21685
I20240503 02:21:48.340739 21807 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:48.458256 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21685
I20240503 02:21:48.480718 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 21554
I20240503 02:21:48.486879 21676 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:48.603513 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 21554
I20240503 02:21:48.627710 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 20534
I20240503 02:21:48.628854 20595 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:48.775033 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 20534
2024-05-03T02:21:48Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (22754 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240503 02:21:48.836120 21353 test_util.cc:268] Using random seed: 1640417537
I20240503 02:21:48.838076 21353 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:21:48.838155 21353 ts_itest-base.cc:113] --------------
I20240503 02:21:48.838222 21353 ts_itest-base.cc:114] 4 tablet servers
I20240503 02:21:48.838271 21353 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:21:48.838351 21353 ts_itest-base.cc:116] --------------
2024-05-03T02:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:21:48Z Disabled control of system clock
I20240503 02:21:48.895557 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:43177
--webserver_interface=127.20.218.126
--webserver_port=0
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.218.126:43177
--location_mapping_cmd=/tmp/dist-test-taskEtZc7o/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240503 02:21:49.250636 23015 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:21:49.251142 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:49.251277 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:49.251483 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:49.259508 23015 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:49.259651 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:21:49.259718 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:49.259768 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:21:49.259862 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:21:49.269570 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskEtZc7o/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.20.218.126:43177
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.218.126:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.218.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:49.271001 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:49.272773 23015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:49.284233 23020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:49.284128 23021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:49.284277 23023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:49.285035 23015 server_base.cc:1034] running on GCE node
I20240503 02:21:49.285857 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:49.287401 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:49.288636 23015 hybrid_clock.cc:648] HybridClock initialized: now 1714702909288575 us; error 39 us; skew 500 ppm
I20240503 02:21:49.288987 23015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:49.291509 23015 webserver.cc:460] Webserver started at http://127.20.218.126:39971/ using document root <none> and password file <none>
I20240503 02:21:49.292279 23015 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:49.292418 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:49.292726 23015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:49.295445 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b7e8875ace254f3e98767491965671b5"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.296233 23015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b7e8875ace254f3e98767491965671b5"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.301725 23015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20240503 02:21:49.305598 23029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:49.306946 23015 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240503 02:21:49.307138 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7e8875ace254f3e98767491965671b5"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.307379 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:49.324755 23015 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:49.325661 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:49.325946 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:49.344748 23015 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.126:43177
I20240503 02:21:49.344795 23080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.126:43177 every 8 connection(s)
I20240503 02:21:49.346215 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:21:49.348373 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23015
I20240503 02:21:49.348590 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:21:49.350273 23081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:49.363746 23081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5: Bootstrap starting.
I20240503 02:21:49.366885 23081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:49.367870 23081 log.cc:824] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:49.370688 23081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5: No bootstrap required, opened a new log
I20240503 02:21:49.377676 23081 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } }
I20240503 02:21:49.378134 23081 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:49.378258 23081 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8875ace254f3e98767491965671b5, State: Initialized, Role: FOLLOWER
I20240503 02:21:49.378893 23081 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } }
I20240503 02:21:49.379129 23081 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:21:49.379267 23081 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:21:49.379446 23081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:49.381755 23081 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } }
I20240503 02:21:49.382269 23081 leader_election.cc:302] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e8875ace254f3e98767491965671b5; no voters: 
I20240503 02:21:49.382871 23081 leader_election.cc:288] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240503 02:21:49.383036 23086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:49.383726 23086 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [term 1 LEADER]: Becoming Leader. State: Replica: b7e8875ace254f3e98767491965671b5, State: Running, Role: LEADER
I20240503 02:21:49.384301 23086 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } }
I20240503 02:21:49.384871 23081 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:21:49.387355 23088 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e8875ace254f3e98767491965671b5. Latest consensus state: current_term: 1 leader_uuid: "b7e8875ace254f3e98767491965671b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } } }
I20240503 02:21:49.387708 23088 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:49.388134 23087 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e8875ace254f3e98767491965671b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e8875ace254f3e98767491965671b5" member_type: VOTER last_known_addr { host: "127.20.218.126" port: 43177 } } }
I20240503 02:21:49.388408 23087 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5 [sys.catalog]: This master's current role is: LEADER
I20240503 02:21:49.389573 23092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:21:49.395797 23092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:21:49.410336 23092 catalog_manager.cc:1347] Generated new cluster ID: e7038f3da42c4834b288bfb322465daf
I20240503 02:21:49.410512 23092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:21:49.442317 23092 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:21:49.444195 23092 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:21:49.464648 23092 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b7e8875ace254f3e98767491965671b5: Generated new TSK 0
I20240503 02:21:49.465646 23092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:21:49.481178 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.65:0
--local_ip_for_outbound_sockets=127.20.218.65
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:21:49.846352 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:49.846673 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:49.846818 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:49.854425 23105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:49.854566 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:49.854732 23105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.65
I20240503 02:21:49.863827 23105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.218.65
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.65
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:49.865190 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:49.866974 23105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:49.878815 23110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:49.879068 23111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:49.879637 23113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:49.880124 23105 server_base.cc:1034] running on GCE node
I20240503 02:21:49.880782 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:49.881754 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:49.882972 23105 hybrid_clock.cc:648] HybridClock initialized: now 1714702909882925 us; error 42 us; skew 500 ppm
I20240503 02:21:49.883308 23105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:49.885632 23105 webserver.cc:460] Webserver started at http://127.20.218.65:35507/ using document root <none> and password file <none>
I20240503 02:21:49.886394 23105 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:49.886504 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:49.886791 23105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:49.891933 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dcbb067767bf4060bbac25f4f5fad61a"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.893061 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dcbb067767bf4060bbac25f4f5fad61a"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.898718 23105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.002s
I20240503 02:21:49.902429 23119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:49.903959 23105 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
I20240503 02:21:49.904148 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dcbb067767bf4060bbac25f4f5fad61a"
format_stamp: "Formatted at 2024-05-03 02:21:49 on dist-test-slave-81lq"
I20240503 02:21:49.904376 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:49.924396 23105 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:49.925292 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:49.925602 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:49.926633 23105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:49.928200 23105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:49.928304 23105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:49.928419 23105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:49.928498 23105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:49.961990 23105 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.65:35687
I20240503 02:21:49.962042 23231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.65:35687 every 8 connection(s)
I20240503 02:21:49.963339 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:21:49.971274 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23105
I20240503 02:21:49.971479 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:21:49.975135 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.66:0
--local_ip_for_outbound_sockets=127.20.218.66
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:49.976708 23232 heartbeater.cc:347] Connected to a master server at 127.20.218.126:43177
I20240503 02:21:49.977094 23232 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:49.978228 23232 heartbeater.cc:511] Master 127.20.218.126:43177 requested a full tablet report, sending...
I20240503 02:21:50.049391 23046 ts_manager.cc:194] Registered new tserver with Master: dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:21:50.051219 23046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.65:57505
W20240503 02:21:50.351588 23236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:50.351917 23236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:50.352066 23236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:50.360952 23236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:50.361115 23236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:50.361284 23236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.66
I20240503 02:21:50.370913 23236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.218.66
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.66
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:50.372313 23236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:50.374140 23236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:50.386390 23245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:50.386372 23242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:50.386600 23243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:50.387287 23236 server_base.cc:1034] running on GCE node
I20240503 02:21:50.387912 23236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:50.388988 23236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:50.390194 23236 hybrid_clock.cc:648] HybridClock initialized: now 1714702910390136 us; error 60 us; skew 500 ppm
I20240503 02:21:50.390576 23236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:50.392983 23236 webserver.cc:460] Webserver started at http://127.20.218.66:36677/ using document root <none> and password file <none>
I20240503 02:21:50.393922 23236 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:50.394064 23236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:50.394378 23236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:50.397101 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.397962 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.403370 23236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240503 02:21:50.406992 23251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.408438 23236 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240503 02:21:50.408629 23236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.408844 23236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:50.439492 23236 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:50.440433 23236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:50.440719 23236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:50.442004 23236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:50.443616 23236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:50.443724 23236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.443845 23236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:50.443919 23236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.476282 23236 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.66:34401
I20240503 02:21:50.476302 23363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.66:34401 every 8 connection(s)
I20240503 02:21:50.477701 23236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:21:50.482152 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23236
I20240503 02:21:50.482317 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:21:50.485760 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.67:0
--local_ip_for_outbound_sockets=127.20.218.67
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:50.491233 23364 heartbeater.cc:347] Connected to a master server at 127.20.218.126:43177
I20240503 02:21:50.491575 23364 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:50.492422 23364 heartbeater.cc:511] Master 127.20.218.126:43177 requested a full tablet report, sending...
I20240503 02:21:50.554070 23046 ts_manager.cc:194] Registered new tserver with Master: b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:21:50.555063 23046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.66:53853
W20240503 02:21:50.851034 23368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:50.851346 23368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:50.851473 23368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:50.859179 23368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:50.859313 23368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:50.859503 23368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.67
I20240503 02:21:50.868686 23368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.218.67
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.67
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:50.870092 23368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:50.871821 23368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:50.883953 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:50.883844 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:50.883875 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:50.884758 23368 server_base.cc:1034] running on GCE node
I20240503 02:21:50.885469 23368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:50.886471 23368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:50.887696 23368 hybrid_clock.cc:648] HybridClock initialized: now 1714702910887644 us; error 50 us; skew 500 ppm
I20240503 02:21:50.888027 23368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:50.890450 23368 webserver.cc:460] Webserver started at http://127.20.218.67:43719/ using document root <none> and password file <none>
I20240503 02:21:50.891229 23368 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:50.891350 23368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:50.891676 23368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:50.894416 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7ec52a420ca6438991468d6d68818019"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.895174 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7ec52a420ca6438991468d6d68818019"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.900470 23368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240503 02:21:50.903887 23383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.905225 23368 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:50.905433 23368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ec52a420ca6438991468d6d68818019"
format_stamp: "Formatted at 2024-05-03 02:21:50 on dist-test-slave-81lq"
I20240503 02:21:50.905644 23368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:50.924690 23368 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:50.925663 23368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:50.925949 23368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:50.926978 23368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:50.928588 23368 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:50.928759 23368 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.928901 23368 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:50.928980 23368 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:50.961802 23368 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.67:41461
I20240503 02:21:50.961899 23495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.67:41461 every 8 connection(s)
I20240503 02:21:50.963109 23368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:21:50.971500 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23368
I20240503 02:21:50.971675 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:21:50.975446 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:0
--local_ip_for_outbound_sockets=127.20.218.68
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:21:50.976578 23496 heartbeater.cc:347] Connected to a master server at 127.20.218.126:43177
I20240503 02:21:50.976897 23496 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:50.977963 23496 heartbeater.cc:511] Master 127.20.218.126:43177 requested a full tablet report, sending...
I20240503 02:21:51.025277 23046 ts_manager.cc:194] Registered new tserver with Master: 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:51.026346 23046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.67:48653
I20240503 02:21:51.054344 23232 heartbeater.cc:503] Master 127.20.218.126:43177 was elected leader, sending a full tablet report...
W20240503 02:21:51.324030 23500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:51.324350 23500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:51.324481 23500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:51.332051 23500 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:51.332185 23500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:51.332358 23500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:51.341884 23500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=0
--tserver_master_addrs=127.20.218.126:43177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:51.343318 23500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:51.345207 23500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:51.357192 23508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:51.357231 23506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:51.357206 23505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:51.358144 23500 server_base.cc:1034] running on GCE node
I20240503 02:21:51.358722 23500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:51.359694 23500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:51.360908 23500 hybrid_clock.cc:648] HybridClock initialized: now 1714702911360847 us; error 43 us; skew 500 ppm
I20240503 02:21:51.361235 23500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:51.363530 23500 webserver.cc:460] Webserver started at http://127.20.218.68:37429/ using document root <none> and password file <none>
I20240503 02:21:51.364267 23500 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:51.364382 23500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:51.364691 23500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:21:51.367357 23500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8f237b4e800e443fae82db47282e606f"
format_stamp: "Formatted at 2024-05-03 02:21:51 on dist-test-slave-81lq"
I20240503 02:21:51.368129 23500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8f237b4e800e443fae82db47282e606f"
format_stamp: "Formatted at 2024-05-03 02:21:51 on dist-test-slave-81lq"
I20240503 02:21:51.373286 23500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240503 02:21:51.376716 23514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:51.378182 23500 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:51.378355 23500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8f237b4e800e443fae82db47282e606f"
format_stamp: "Formatted at 2024-05-03 02:21:51 on dist-test-slave-81lq"
I20240503 02:21:51.378563 23500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:51.400355 23500 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:51.401242 23500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:51.401533 23500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:51.402546 23500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:51.404042 23500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:51.404130 23500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:51.404248 23500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:51.404320 23500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:51.437630 23500 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:41297
I20240503 02:21:51.437690 23627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:41297 every 8 connection(s)
I20240503 02:21:51.439029 23500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:51.448956 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23500
I20240503 02:21:51.449149 21353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:21:51.452075 23628 heartbeater.cc:347] Connected to a master server at 127.20.218.126:43177
I20240503 02:21:51.452435 23628 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:51.453527 23628 heartbeater.cc:511] Master 127.20.218.126:43177 requested a full tablet report, sending...
I20240503 02:21:51.502234 23046 ts_manager.cc:194] Registered new tserver with Master: 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297)
I20240503 02:21:51.503409 23046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:33111
I20240503 02:21:51.504137 21353 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:21:51.521512 21353 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240503 02:21:51.521698 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23500
I20240503 02:21:51.530354 23623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:21:51.557957 23364 heartbeater.cc:503] Master 127.20.218.126:43177 was elected leader, sending a full tablet report...
I20240503 02:21:51.612265 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23500
I20240503 02:21:52.029100 23496 heartbeater.cc:503] Master 127.20.218.126:43177 was elected leader, sending a full tablet report...
I20240503 02:21:55.381840 23046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60706:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:55.388829 23046 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:55.421782 23430 tablet_service.cc:1467] Processing CreateTablet for tablet 50bb8de00f2046c3b4ff426bbc87b37f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.421257 23298 tablet_service.cc:1467] Processing CreateTablet for tablet 50bb8de00f2046c3b4ff426bbc87b37f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.421252 23166 tablet_service.cc:1467] Processing CreateTablet for tablet 50bb8de00f2046c3b4ff426bbc87b37f (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.422472 23429 tablet_service.cc:1467] Processing CreateTablet for tablet 100904c00d104bcca2956b21edf2c496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.422859 23430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bb8de00f2046c3b4ff426bbc87b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.422853 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bb8de00f2046c3b4ff426bbc87b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.422852 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bb8de00f2046c3b4ff426bbc87b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.423319 23297 tablet_service.cc:1467] Processing CreateTablet for tablet 100904c00d104bcca2956b21edf2c496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.423790 23297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100904c00d104bcca2956b21edf2c496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.424822 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100904c00d104bcca2956b21edf2c496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.425537 23165 tablet_service.cc:1467] Processing CreateTablet for tablet 100904c00d104bcca2956b21edf2c496 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.425763 23167 tablet_service.cc:1467] Processing CreateTablet for tablet a5e1df757e4f40559ab9a3d5b9291550 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.426064 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100904c00d104bcca2956b21edf2c496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.426443 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e1df757e4f40559ab9a3d5b9291550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.427081 23299 tablet_service.cc:1467] Processing CreateTablet for tablet a5e1df757e4f40559ab9a3d5b9291550 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.427395 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e1df757e4f40559ab9a3d5b9291550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.421257 23431 tablet_service.cc:1467] Processing CreateTablet for tablet a5e1df757e4f40559ab9a3d5b9291550 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:55.430022 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e1df757e4f40559ab9a3d5b9291550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:55.446581 23650 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:55.450537 23650 tablet_bootstrap.cc:654] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.451586 23650 log.cc:824] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:55.452548 23652 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:55.454587 23651 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:55.457087 23651 tablet_bootstrap.cc:654] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.457269 23652 tablet_bootstrap.cc:654] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.458405 23651 log.cc:824] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:55.458484 23652 log.cc:824] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Log is configured to *not* fsync() on all Append() calls
I20240503 02:21:55.460196 23650 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:55.460574 23650 ts_tablet_manager.cc:1367] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.006s
I20240503 02:21:55.460880 23652 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:55.461122 23652 ts_tablet_manager.cc:1367] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 02:21:55.461197 23651 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:55.461493 23651 ts_tablet_manager.cc:1367] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.007s	user 0.007s	sys 0.000s
I20240503 02:21:55.466979 23652 raft_consensus.cc:348] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.467398 23652 raft_consensus.cc:374] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.467523 23652 raft_consensus.cc:729] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.468214 23652 consensus_queue.cc:260] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.469946 23652 ts_tablet_manager.cc:1398] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20240503 02:21:55.470383 23652 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:55.470294 23651 raft_consensus.cc:348] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.470762 23651 raft_consensus.cc:374] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.470885 23651 raft_consensus.cc:729] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.471295 23650 raft_consensus.cc:348] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.471714 23650 raft_consensus.cc:374] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.471714 23651 consensus_queue.cc:260] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.471832 23650 raft_consensus.cc:729] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.472093 23652 tablet_bootstrap.cc:654] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.472651 23650 consensus_queue.cc:260] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.479310 23650 ts_tablet_manager.cc:1398] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.019s	user 0.014s	sys 0.000s
I20240503 02:21:55.479827 23650 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:55.479952 23652 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:55.480149 23652 ts_tablet_manager.cc:1367] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240503 02:21:55.481443 23652 raft_consensus.cc:348] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.481714 23652 raft_consensus.cc:374] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.481843 23652 raft_consensus.cc:729] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.482100 23652 consensus_queue.cc:260] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.482599 23652 ts_tablet_manager.cc:1398] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:21:55.482894 23652 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
W20240503 02:21:55.483456 23365 tablet.cc:2369] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:21:55.484071 23650 tablet_bootstrap.cc:654] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.484634 23652 tablet_bootstrap.cc:654] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.491057 23651 ts_tablet_manager.cc:1398] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.004s
I20240503 02:21:55.491786 23651 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:55.494273 23651 tablet_bootstrap.cc:654] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.497020 23650 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:55.497241 23650 ts_tablet_manager.cc:1367] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.003s
I20240503 02:21:55.497931 23652 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:55.498225 23652 ts_tablet_manager.cc:1367] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.008s
I20240503 02:21:55.498390 23650 raft_consensus.cc:348] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.498679 23650 raft_consensus.cc:374] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.498823 23650 raft_consensus.cc:729] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.499148 23652 raft_consensus.cc:348] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.499169 23650 consensus_queue.cc:260] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.499341 23652 raft_consensus.cc:374] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.499441 23652 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.499729 23652 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.499995 23650 ts_tablet_manager.cc:1398] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:55.500057 23651 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:55.500276 23650 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:55.500478 23651 ts_tablet_manager.cc:1367] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240503 02:21:55.501221 23652 ts_tablet_manager.cc:1398] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240503 02:21:55.501585 23651 raft_consensus.cc:348] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.501852 23651 raft_consensus.cc:374] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.502010 23651 raft_consensus.cc:729] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.502281 23651 consensus_queue.cc:260] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.502511 23650 tablet_bootstrap.cc:654] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.502964 23651 ts_tablet_manager.cc:1398] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:55.503285 23651 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:55.504911 23651 tablet_bootstrap.cc:654] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:55.506485 23650 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:55.506716 23650 ts_tablet_manager.cc:1367] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240503 02:21:55.506843 23651 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:55.507035 23651 ts_tablet_manager.cc:1367] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240503 02:21:55.507464 23650 raft_consensus.cc:348] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.507705 23650 raft_consensus.cc:374] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.507800 23650 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.507746 23651 raft_consensus.cc:348] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.508008 23651 raft_consensus.cc:374] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:55.508102 23651 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:55.508057 23650 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.508378 23651 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.508817 23650 ts_tablet_manager.cc:1398] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:55.508971 23651 ts_tablet_manager.cc:1398] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:21:55.537047 23659 raft_consensus.cc:482] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:55.537313 23659 raft_consensus.cc:504] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.539074 23659 leader_election.cc:288] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.546283 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bb8de00f2046c3b4ff426bbc87b37f" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:55.546283 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bb8de00f2046c3b4ff426bbc87b37f" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:55.546774 23451 raft_consensus.cc:2454] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.546787 23319 raft_consensus.cc:2454] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.547379 23123 leader_election.cc:302] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.547787 23659 raft_consensus.cc:2781] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:55.547943 23659 raft_consensus.cc:482] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:55.548053 23659 raft_consensus.cc:3037] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.550174 23659 raft_consensus.cc:504] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.550771 23659 leader_election.cc:288] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.551302 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bb8de00f2046c3b4ff426bbc87b37f" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:55.551324 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50bb8de00f2046c3b4ff426bbc87b37f" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:55.551497 23319 raft_consensus.cc:3037] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.551498 23451 raft_consensus.cc:3037] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.553649 23451 raft_consensus.cc:2454] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.553650 23319 raft_consensus.cc:2454] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.554107 23120 leader_election.cc:302] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.554430 23659 raft_consensus.cc:2781] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:55.554903 23659 raft_consensus.cc:686] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:21:55.555480 23659 consensus_queue.cc:237] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.560122 23044 catalog_manager.cc:5484] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 0 to 1, leader changed from <none> to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:55.621208 23659 raft_consensus.cc:482] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:55.621497 23659 raft_consensus.cc:504] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.622176 23659 leader_election.cc:288] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.622798 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:55.622807 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:55.623061 23319 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.623080 23451 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.623512 23123 leader_election.cc:302] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.623821 23659 raft_consensus.cc:2781] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:55.623924 23659 raft_consensus.cc:482] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:55.624176 23659 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.626297 23659 raft_consensus.cc:504] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.626966 23659 leader_election.cc:288] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.627534 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:55.627537 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:55.627756 23319 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.627756 23451 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.629685 23451 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.629703 23319 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.630142 23120 leader_election.cc:302] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.630434 23659 raft_consensus.cc:2781] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:55.630604 23659 raft_consensus.cc:686] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:21:55.630905 23659 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.633790 23044 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 0 to 1, leader changed from <none> to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:55.680290 23659 raft_consensus.cc:482] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:55.680508 23659 raft_consensus.cc:504] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.681324 23659 leader_election.cc:288] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.681885 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e1df757e4f40559ab9a3d5b9291550" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:55.681960 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e1df757e4f40559ab9a3d5b9291550" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:55.682133 23319 raft_consensus.cc:2454] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.682168 23451 raft_consensus.cc:2454] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:55.682595 23120 leader_election.cc:302] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.682868 23659 raft_consensus.cc:2781] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:55.682977 23659 raft_consensus.cc:482] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:55.683049 23659 raft_consensus.cc:3037] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.684895 23659 raft_consensus.cc:504] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.685547 23659 leader_election.cc:288] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:55.686091 23451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e1df757e4f40559ab9a3d5b9291550" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:55.686116 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e1df757e4f40559ab9a3d5b9291550" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:55.686280 23319 raft_consensus.cc:3037] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.686288 23451 raft_consensus.cc:3037] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:55.688149 23319 raft_consensus.cc:2454] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.688509 23451 raft_consensus.cc:2454] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:55.688563 23120 leader_election.cc:302] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:55.689042 23659 raft_consensus.cc:2781] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:55.689217 23659 raft_consensus.cc:686] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:21:55.689594 23659 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:55.692412 23044 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 0 to 1, leader changed from <none> to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
W20240503 02:21:55.718528 23497 tablet.cc:2369] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:21:55.718884 23233 tablet.cc:2369] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:21:56.028432 23659 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:56.052047 23667 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:56.088586 23671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:56.088896 23671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:56.096890 23671 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:21:56.100764 23667 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:56.109926 23659 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:21:56.156436 23659 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:56.164475 23659 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 345.301 ms
  time per row: 0.172651 ms
I20240503 02:21:56.624212 23044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60706:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:56.625033 23044 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:56.633523 23297 tablet_service.cc:1467] Processing CreateTablet for tablet bb8aafe8adda4229bbe493e76dc8aed4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.633705 23166 tablet_service.cc:1467] Processing CreateTablet for tablet bb8aafe8adda4229bbe493e76dc8aed4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.634030 23297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8aafe8adda4229bbe493e76dc8aed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.634158 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8aafe8adda4229bbe493e76dc8aed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.635149 23431 tablet_service.cc:1467] Processing CreateTablet for tablet bb8aafe8adda4229bbe493e76dc8aed4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.635324 23299 tablet_service.cc:1467] Processing CreateTablet for tablet 5db050761e584724a0710ed437a0bb32 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.635578 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8aafe8adda4229bbe493e76dc8aed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.635687 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db050761e584724a0710ed437a0bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.637792 23430 tablet_service.cc:1467] Processing CreateTablet for tablet 5db050761e584724a0710ed437a0bb32 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.638161 23430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db050761e584724a0710ed437a0bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.641849 23298 tablet_service.cc:1467] Processing CreateTablet for tablet beb46d2ddb004a6d8498000a326f2d29 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.641892 23429 tablet_service.cc:1467] Processing CreateTablet for tablet beb46d2ddb004a6d8498000a326f2d29 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.642354 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb46d2ddb004a6d8498000a326f2d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.642374 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb46d2ddb004a6d8498000a326f2d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.646569 23165 tablet_service.cc:1467] Processing CreateTablet for tablet 5db050761e584724a0710ed437a0bb32 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.647128 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db050761e584724a0710ed437a0bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.649901 23167 tablet_service.cc:1467] Processing CreateTablet for tablet beb46d2ddb004a6d8498000a326f2d29 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:56.650439 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb46d2ddb004a6d8498000a326f2d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:56.653868 23732 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:56.655750 23732 tablet_bootstrap.cc:654] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.661492 23732 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:56.661883 23732 ts_tablet_manager.cc:1367] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.007s
I20240503 02:21:56.663343 23732 raft_consensus.cc:348] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.663609 23732 raft_consensus.cc:374] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.663741 23732 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.664054 23732 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.664644 23732 ts_tablet_manager.cc:1398] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:56.673192 23732 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:56.673918 23731 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:56.674026 23734 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:56.675273 23732 tablet_bootstrap.cc:654] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.675427 23734 tablet_bootstrap.cc:654] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.675825 23731 tablet_bootstrap.cc:654] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.677522 23734 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:56.677665 23734 ts_tablet_manager.cc:1367] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240503 02:21:56.678148 23732 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:56.678321 23732 ts_tablet_manager.cc:1367] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240503 02:21:56.678531 23731 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:56.678670 23731 ts_tablet_manager.cc:1367] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240503 02:21:56.678552 23734 raft_consensus.cc:348] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.678793 23734 raft_consensus.cc:374] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.678882 23734 raft_consensus.cc:729] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.679118 23732 raft_consensus.cc:348] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.679117 23734 consensus_queue.cc:260] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.679347 23732 raft_consensus.cc:374] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.679316 23731 raft_consensus.cc:348] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.679419 23732 raft_consensus.cc:729] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.679473 23731 raft_consensus.cc:374] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.679553 23731 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.679709 23732 consensus_queue.cc:260] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.679864 23731 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.680320 23732 ts_tablet_manager.cc:1398] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:56.680534 23731 ts_tablet_manager.cc:1398] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:56.680634 23732 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:56.680836 23731 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:56.682155 23734 ts_tablet_manager.cc:1398] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:21:56.682415 23732 tablet_bootstrap.cc:654] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.682474 23734 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:56.685140 23731 tablet_bootstrap.cc:654] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.687001 23732 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:56.687256 23732 ts_tablet_manager.cc:1367] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.002s
I20240503 02:21:56.688416 23734 tablet_bootstrap.cc:654] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.688387 23732 raft_consensus.cc:348] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.688664 23732 raft_consensus.cc:374] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.688782 23732 raft_consensus.cc:729] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.689088 23732 consensus_queue.cc:260] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.690032 23732 ts_tablet_manager.cc:1398] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240503 02:21:56.690925 23731 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:56.690932 23734 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:56.691108 23734 ts_tablet_manager.cc:1367] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.002s
I20240503 02:21:56.691155 23731 ts_tablet_manager.cc:1367] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.004s
I20240503 02:21:56.691923 23734 raft_consensus.cc:348] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.691923 23731 raft_consensus.cc:348] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.692154 23731 raft_consensus.cc:374] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.692154 23734 raft_consensus.cc:374] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.692232 23734 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.692227 23731 raft_consensus.cc:729] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.692498 23734 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.692510 23731 consensus_queue.cc:260] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.693022 23731 ts_tablet_manager.cc:1398] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:21:56.693048 23734 ts_tablet_manager.cc:1398] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240503 02:21:56.693303 23734 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:56.693303 23731 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:56.694953 23731 tablet_bootstrap.cc:654] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.694962 23734 tablet_bootstrap.cc:654] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:56.697446 23734 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:56.697599 23734 ts_tablet_manager.cc:1367] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240503 02:21:56.697912 23731 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:56.698074 23731 ts_tablet_manager.cc:1367] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:21:56.698382 23734 raft_consensus.cc:348] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.698586 23734 raft_consensus.cc:374] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.698685 23734 raft_consensus.cc:729] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.698843 23731 raft_consensus.cc:348] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.699051 23731 raft_consensus.cc:374] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:56.698938 23734 consensus_queue.cc:260] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.699144 23731 raft_consensus.cc:729] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:56.699368 23731 consensus_queue.cc:260] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.699503 23734 ts_tablet_manager.cc:1398] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:56.700008 23731 ts_tablet_manager.cc:1398] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:56.728592 23738 raft_consensus.cc:482] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:56.728790 23738 raft_consensus.cc:504] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.730479 23738 leader_election.cc:288] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:21:56.738042 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:56.738111 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a" is_pre_election: true
I20240503 02:21:56.738409 23318 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 0.
I20240503 02:21:56.738541 23187 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 0.
I20240503 02:21:56.738976 23384 leader_election.cc:302] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, b0c0b6bc47954db9ada48f0c09ee9959; no voters: 
I20240503 02:21:56.739372 23738 raft_consensus.cc:2781] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:56.739491 23738 raft_consensus.cc:482] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:56.739586 23738 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.741577 23738 raft_consensus.cc:504] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.742236 23738 leader_election.cc:288] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 election: Requested vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:21:56.742624 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:21:56.742756 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:56.742837 23187 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.742964 23318 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.745906 23187 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 1.
I20240503 02:21:56.745906 23318 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 1.
I20240503 02:21:56.746340 23384 leader_election.cc:302] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, b0c0b6bc47954db9ada48f0c09ee9959; no voters: 
I20240503 02:21:56.746673 23738 raft_consensus.cc:2781] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:56.747112 23738 raft_consensus.cc:686] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Running, Role: LEADER
I20240503 02:21:56.747629 23738 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.750921 23039 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec52a420ca6438991468d6d68818019 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } }
I20240503 02:21:56.812323 23672 raft_consensus.cc:482] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:56.812582 23672 raft_consensus.cc:504] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.814275 23672 leader_election.cc:288] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:56.822883 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db050761e584724a0710ed437a0bb32" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a" is_pre_election: true
I20240503 02:21:56.823165 23187 raft_consensus.cc:2454] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 0.
I20240503 02:21:56.823639 23255 leader_election.cc:302] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:56.824038 23672 raft_consensus.cc:2781] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:56.824151 23672 raft_consensus.cc:482] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:56.824256 23672 raft_consensus.cc:3037] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.824353 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db050761e584724a0710ed437a0bb32" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:56.824579 23450 raft_consensus.cc:2454] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 0.
I20240503 02:21:56.826524 23672 raft_consensus.cc:504] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.827189 23672 leader_election.cc:288] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 election: Requested vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:56.827720 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db050761e584724a0710ed437a0bb32" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:21:56.827754 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db050761e584724a0710ed437a0bb32" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:56.827926 23450 raft_consensus.cc:3037] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.827963 23187 raft_consensus.cc:3037] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.829911 23450 raft_consensus.cc:2454] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 1.
I20240503 02:21:56.829921 23187 raft_consensus.cc:2454] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 1.
I20240503 02:21:56.830358 23255 leader_election.cc:302] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:56.830698 23672 raft_consensus.cc:2781] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:56.831136 23672 raft_consensus.cc:686] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Becoming Leader. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Running, Role: LEADER
I20240503 02:21:56.831622 23672 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.835156 23039 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:56.901075 23738 raft_consensus.cc:482] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:56.901382 23738 raft_consensus.cc:504] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.902081 23738 leader_election.cc:288] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:21:56.902770 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a" is_pre_election: true
I20240503 02:21:56.902750 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:56.903048 23318 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 0.
I20240503 02:21:56.903055 23187 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 0.
I20240503 02:21:56.903508 23384 leader_election.cc:302] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, b0c0b6bc47954db9ada48f0c09ee9959; no voters: 
I20240503 02:21:56.903806 23738 raft_consensus.cc:2781] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:56.903900 23738 raft_consensus.cc:482] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:56.904013 23738 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.906015 23738 raft_consensus.cc:504] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.906622 23738 leader_election.cc:288] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:21:56.907099 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:56.907150 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "7ec52a420ca6438991468d6d68818019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:21:56.907337 23318 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.907337 23187 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:56.909507 23318 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 1.
I20240503 02:21:56.909502 23187 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec52a420ca6438991468d6d68818019 in term 1.
I20240503 02:21:56.909952 23384 leader_election.cc:302] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, b0c0b6bc47954db9ada48f0c09ee9959; no voters: 
I20240503 02:21:56.910250 23738 raft_consensus.cc:2781] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:56.910409 23738 raft_consensus.cc:686] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Running, Role: LEADER
I20240503 02:21:56.910708 23738 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:56.913677 23039 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec52a420ca6438991468d6d68818019 (127.20.218.67). New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } }
I20240503 02:21:57.210999 23738 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:57.222868 23738 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:57.227602 23672 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:57.245298 23672 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:57.379093 23755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:57.379437 23755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:57.387460 23755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 02:21:57.450649 23738 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:57.462761 23747 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:57.515802 23777 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.20.218.126:43177, user_credentials={real_user=slave}} blocked reactor thread for 47431.5us
W20240503 02:21:57.534283 23785 meta_cache.cc:1261] Time spent looking up entry by key: real 0.066s	user 0.001s	sys 0.000s

INSERT report
    rows total: 2000
    time total: 335.999 ms
  time per row: 0.168 ms
I20240503 02:21:57.888636 23039 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60706:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240503 02:21:57.889398 23039 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 02:21:57.899258 23166 tablet_service.cc:1467] Processing CreateTablet for tablet 61e4ce4f9e734b93b86d351eb411050c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.899530 23297 tablet_service.cc:1467] Processing CreateTablet for tablet 61e4ce4f9e734b93b86d351eb411050c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.899763 23165 tablet_service.cc:1467] Processing CreateTablet for tablet 54227bfcbf9c4ad193075e5eec20a8ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.899899 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e4ce4f9e734b93b86d351eb411050c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.900022 23297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e4ce4f9e734b93b86d351eb411050c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.900177 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54227bfcbf9c4ad193075e5eec20a8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.901757 23431 tablet_service.cc:1467] Processing CreateTablet for tablet 11e7201491ab41beb241006b9ce12624 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.902194 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e7201491ab41beb241006b9ce12624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.905830 23430 tablet_service.cc:1467] Processing CreateTablet for tablet 61e4ce4f9e734b93b86d351eb411050c (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.906282 23430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e4ce4f9e734b93b86d351eb411050c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.909065 23429 tablet_service.cc:1467] Processing CreateTablet for tablet 54227bfcbf9c4ad193075e5eec20a8ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.909665 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54227bfcbf9c4ad193075e5eec20a8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.910287 23817 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:57.913131 23817 tablet_bootstrap.cc:654] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.913745 23167 tablet_service.cc:1467] Processing CreateTablet for tablet 11e7201491ab41beb241006b9ce12624 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.914171 23167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e7201491ab41beb241006b9ce12624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.916203 23298 tablet_service.cc:1467] Processing CreateTablet for tablet 54227bfcbf9c4ad193075e5eec20a8ad (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.916663 23298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54227bfcbf9c4ad193075e5eec20a8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.917738 23299 tablet_service.cc:1467] Processing CreateTablet for tablet 11e7201491ab41beb241006b9ce12624 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:21:57.918190 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e7201491ab41beb241006b9ce12624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:21:57.925091 23816 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:57.928211 23816 tablet_bootstrap.cc:654] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.932637 23817 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:57.932832 23817 ts_tablet_manager.cc:1367] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.005s
I20240503 02:21:57.933871 23818 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:57.933828 23817 raft_consensus.cc:348] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.934154 23817 raft_consensus.cc:374] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.934268 23817 raft_consensus.cc:729] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.934576 23817 consensus_queue.cc:260] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.934713 23816 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:57.934882 23816 ts_tablet_manager.cc:1367] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240503 02:21:57.935178 23817 ts_tablet_manager.cc:1398] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:57.935526 23817 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:57.935945 23818 tablet_bootstrap.cc:654] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.937297 23817 tablet_bootstrap.cc:654] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.937639 23816 raft_consensus.cc:348] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.937945 23816 raft_consensus.cc:374] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.938052 23816 raft_consensus.cc:729] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.938354 23816 consensus_queue.cc:260] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.939193 23816 ts_tablet_manager.cc:1398] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240503 02:21:57.939589 23816 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:57.942423 23816 tablet_bootstrap.cc:654] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.945734 23817 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:57.945920 23817 ts_tablet_manager.cc:1367] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240503 02:21:57.946811 23817 raft_consensus.cc:348] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.947088 23817 raft_consensus.cc:374] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.947166 23817 raft_consensus.cc:729] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.947461 23817 consensus_queue.cc:260] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.948145 23817 ts_tablet_manager.cc:1398] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:57.948498 23817 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Bootstrap starting.
I20240503 02:21:57.950290 23817 tablet_bootstrap.cc:654] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.951330 23816 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:57.951534 23816 ts_tablet_manager.cc:1367] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.004s
I20240503 02:21:57.952301 23818 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:57.952469 23818 ts_tablet_manager.cc:1367] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.019s	user 0.000s	sys 0.016s
I20240503 02:21:57.952428 23816 raft_consensus.cc:348] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.952669 23816 raft_consensus.cc:374] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.952754 23816 raft_consensus.cc:729] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.953194 23817 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: No bootstrap required, opened a new log
I20240503 02:21:57.953126 23816 consensus_queue.cc:260] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.953358 23818 raft_consensus.cc:348] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.953565 23818 raft_consensus.cc:374] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.953649 23818 raft_consensus.cc:729] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.953727 23816 ts_tablet_manager.cc:1398] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240503 02:21:57.953886 23817 ts_tablet_manager.cc:1367] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240503 02:21:57.953879 23818 consensus_queue.cc:260] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.954488 23818 ts_tablet_manager.cc:1398] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:57.954869 23818 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:57.955184 23816 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:21:57.956549 23818 tablet_bootstrap.cc:654] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.956506 23817 raft_consensus.cc:348] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.956841 23817 raft_consensus.cc:374] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.957011 23817 raft_consensus.cc:729] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.957317 23817 consensus_queue.cc:260] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.958101 23817 ts_tablet_manager.cc:1398] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240503 02:21:57.960029 23816 tablet_bootstrap.cc:654] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.962247 23818 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:57.962260 23816 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: No bootstrap required, opened a new log
I20240503 02:21:57.962435 23818 ts_tablet_manager.cc:1367] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.002s
I20240503 02:21:57.962435 23816 ts_tablet_manager.cc:1367] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240503 02:21:57.963256 23816 raft_consensus.cc:348] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.963238 23818 raft_consensus.cc:348] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.963475 23816 raft_consensus.cc:374] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.963490 23818 raft_consensus.cc:374] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.963578 23818 raft_consensus.cc:729] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.963598 23816 raft_consensus.cc:729] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.963824 23816 consensus_queue.cc:260] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.963841 23818 consensus_queue.cc:260] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.964385 23816 ts_tablet_manager.cc:1398] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:57.964407 23818 ts_tablet_manager.cc:1398] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240503 02:21:57.964864 23818 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:21:57.966432 23818 tablet_bootstrap.cc:654] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Neither blocks nor log segments found. Creating new log.
I20240503 02:21:57.982062 23818 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: No bootstrap required, opened a new log
I20240503 02:21:57.982246 23818 ts_tablet_manager.cc:1367] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.017s	user 0.004s	sys 0.011s
I20240503 02:21:57.983078 23818 raft_consensus.cc:348] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.983314 23818 raft_consensus.cc:374] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:21:57.983412 23818 raft_consensus.cc:729] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: FOLLOWER
I20240503 02:21:57.983662 23818 consensus_queue.cc:260] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:57.984359 23818 ts_tablet_manager.cc:1398] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:21:58.068751 23672 raft_consensus.cc:482] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:58.069002 23672 raft_consensus.cc:504] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.069692 23672 leader_election.cc:288] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.070399 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e7201491ab41beb241006b9ce12624" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a" is_pre_election: true
I20240503 02:21:58.070475 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e7201491ab41beb241006b9ce12624" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:58.070647 23186 raft_consensus.cc:2454] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 0.
I20240503 02:21:58.070690 23450 raft_consensus.cc:2454] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 0.
I20240503 02:21:58.071213 23255 leader_election.cc:302] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:58.071556 23672 raft_consensus.cc:2781] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:58.071671 23672 raft_consensus.cc:482] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:58.071782 23672 raft_consensus.cc:3037] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.073745 23672 raft_consensus.cc:504] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.074302 23672 leader_election.cc:288] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 election: Requested vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.074785 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e7201491ab41beb241006b9ce12624" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:21:58.074796 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e7201491ab41beb241006b9ce12624" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:58.074977 23186 raft_consensus.cc:3037] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.074991 23450 raft_consensus.cc:3037] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.077183 23186 raft_consensus.cc:2454] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 1.
I20240503 02:21:58.077209 23450 raft_consensus.cc:2454] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 1.
I20240503 02:21:58.077666 23255 leader_election.cc:302] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, b0c0b6bc47954db9ada48f0c09ee9959; no voters: 
I20240503 02:21:58.077952 23672 raft_consensus.cc:2781] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:58.078119 23672 raft_consensus.cc:686] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Becoming Leader. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Running, Role: LEADER
I20240503 02:21:58.078406 23672 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.081182 23039 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66). New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:58.094802 23659 raft_consensus.cc:482] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:58.095006 23659 raft_consensus.cc:504] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.095654 23659 leader_election.cc:288] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.096184 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e4ce4f9e734b93b86d351eb411050c" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:58.096225 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e4ce4f9e734b93b86d351eb411050c" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:58.096415 23319 raft_consensus.cc:2454] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:58.096446 23450 raft_consensus.cc:2454] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:58.096892 23123 leader_election.cc:302] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:58.097208 23659 raft_consensus.cc:2781] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:58.097297 23659 raft_consensus.cc:482] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:58.097412 23659 raft_consensus.cc:3037] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.099366 23659 raft_consensus.cc:504] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.099979 23659 leader_election.cc:288] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.100454 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e4ce4f9e734b93b86d351eb411050c" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:58.100459 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61e4ce4f9e734b93b86d351eb411050c" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:58.100668 23319 raft_consensus.cc:3037] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.100654 23450 raft_consensus.cc:3037] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.102643 23319 raft_consensus.cc:2454] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:58.102654 23450 raft_consensus.cc:2454] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:58.103088 23123 leader_election.cc:302] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:58.103379 23659 raft_consensus.cc:2781] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:58.103765 23659 raft_consensus.cc:686] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:21:58.104079 23659 consensus_queue.cc:237] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.106895 23039 catalog_manager.cc:5484] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 0 to 1, leader changed from <none> to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:58.177114 23659 raft_consensus.cc:482] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:21:58.177358 23659 raft_consensus.cc:504] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.178221 23659 leader_election.cc:288] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.178731 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54227bfcbf9c4ad193075e5eec20a8ad" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019" is_pre_election: true
I20240503 02:21:58.178822 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54227bfcbf9c4ad193075e5eec20a8ad" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" is_pre_election: true
I20240503 02:21:58.178962 23450 raft_consensus.cc:2454] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:58.179072 23319 raft_consensus.cc:2454] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 0.
I20240503 02:21:58.179438 23123 leader_election.cc:302] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:58.179805 23659 raft_consensus.cc:2781] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:21:58.179906 23659 raft_consensus.cc:482] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:21:58.180039 23659 raft_consensus.cc:3037] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.182091 23659 raft_consensus.cc:504] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.182734 23659 leader_election.cc:288] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Requested vote from peers b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:21:58.183324 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54227bfcbf9c4ad193075e5eec20a8ad" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:21:58.183336 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54227bfcbf9c4ad193075e5eec20a8ad" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:21:58.183503 23319 raft_consensus.cc:3037] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.183543 23450 raft_consensus.cc:3037] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:21:58.185621 23450 raft_consensus.cc:2454] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:58.185642 23319 raft_consensus.cc:2454] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 1.
I20240503 02:21:58.186091 23120 leader_election.cc:302] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:21:58.186455 23659 raft_consensus.cc:2781] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:21:58.186589 23659 raft_consensus.cc:686] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:21:58.186893 23659 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } }
I20240503 02:21:58.189723 23039 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 0 to 1, leader changed from <none> to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } }
I20240503 02:21:58.552750 23672 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:58.558674 23672 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:21:58.597936 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:58.598300 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:58.605973 23832 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:21:58.612077 23659 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:58.616724 23659 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 02:21:58.683292 23829 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:21:58.696348 23829 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 255.773 ms
  time per row: 0.127886 ms
I20240503 02:21:59.017910 21353 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
/tmp/dist-test-taskEtZc7o/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.218.68:41297
--local_ip_for_outbound_sockets=127.20.218.68
--tserver_master_addrs=127.20.218.126:43177
--webserver_port=37429
--webserver_interface=127.20.218.68
--builtin_ntp_servers=127.20.218.84:40789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:21:59.377097 23895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:21:59.377625 23895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:21:59.377806 23895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:21:59.386605 23895 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:21:59.386772 23895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:21:59.386919 23895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.218.68
I20240503 02:21:59.396524 23895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.218.84:40789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.218.68:41297
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.218.68
--webserver_port=37429
--tserver_master_addrs=127.20.218.126:43177
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.218.68
--log_dir=/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:21:59.397969 23895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:21:59.399720 23895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:21:59.412521 23900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:59.412483 23901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:21:59.412451 23903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:21:59.413228 23895 server_base.cc:1034] running on GCE node
I20240503 02:21:59.413928 23895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:21:59.415000 23895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:21:59.416247 23895 hybrid_clock.cc:648] HybridClock initialized: now 1714702919416185 us; error 54 us; skew 500 ppm
I20240503 02:21:59.416591 23895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:21:59.419080 23895 webserver.cc:460] Webserver started at http://127.20.218.68:37429/ using document root <none> and password file <none>
I20240503 02:21:59.419934 23895 fs_manager.cc:359] Metadata directory not provided
I20240503 02:21:59.420084 23895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:21:59.425648 23895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:59.428382 23909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:59.429998 23895 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240503 02:21:59.430173 23895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8f237b4e800e443fae82db47282e606f"
format_stamp: "Formatted at 2024-05-03 02:21:51 on dist-test-slave-81lq"
I20240503 02:21:59.431175 23895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:21:59.452545 23895 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240503 02:21:59.453462 23895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:21:59.453756 23895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:21:59.454752 23895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:21:59.456393 23895 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:21:59.456475 23895 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:59.456625 23895 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:21:59.456696 23895 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240503 02:21:59.490582 23895 rpc_server.cc:307] RPC server started. Bound to: 127.20.218.68:41297
I20240503 02:21:59.490612 24021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.218.68:41297 every 8 connection(s)
I20240503 02:21:59.492434 23895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:21:59.493134 21353 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu as pid 23895
I20240503 02:21:59.518874 24022 heartbeater.cc:347] Connected to a master server at 127.20.218.126:43177
I20240503 02:21:59.519199 24022 heartbeater.cc:464] Registering TS with master...
I20240503 02:21:59.520081 24022 heartbeater.cc:511] Master 127.20.218.126:43177 requested a full tablet report, sending...
I20240503 02:21:59.522539 23037 ts_manager.cc:194] Re-registered known tserver with Master: 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297)
I20240503 02:21:59.523818 23037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.218.68:55319
I20240503 02:22:00.043437 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:00.044939 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.047809 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:00.060829 23297 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.228173 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:00.228655 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.236802 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:00.241962 23297 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.429576 23297 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.429920 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:00.431968 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.432004 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:00.526855 24022 heartbeater.cc:503] Master 127.20.218.126:43177 was elected leader, sending a full tablet report...
I20240503 02:22:00.619093 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.619483 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:00.622251 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:00.623013 23297 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:00.704668 23186 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:00.708071 23319 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.709158 24166 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.709604 23450 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.710785 24166 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.713485 23829 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.714215 23319 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.716132 23449 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
W20240503 02:22:00.721125 23123 consensus_peers.cc:485] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54227bfcbf9c4ad193075e5eec20a8ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:00.722579 23045 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:00.738330 23186 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:00.741178 23449 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.741284 23319 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:22:00.741415 23123 consensus_peers.cc:485] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 100904c00d104bcca2956b21edf2c496. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:00.741848 24128 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.742167 24166 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.745075 24166 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.745885 23319 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.746503 23449 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.750612 23043 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:00.768631 23449 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:00.772111 23319 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.772694 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.772840 24188 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.773492 23802 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.776688 24188 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.777709 23319 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.782208 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.784241 23037 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:22:00.794377 23387 consensus_peers.cc:485] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb8aafe8adda4229bbe493e76dc8aed4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:00.806653 23319 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:00.810920 23186 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.810987 23449 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.811904 24200 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.812379 23672 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.816720 24200 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.817633 23449 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
W20240503 02:22:00.818367 23255 consensus_peers.cc:485] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5db050761e584724a0710ed437a0bb32. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:00.819041 23186 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.823192 23037 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.839737 23186 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
W20240503 02:22:00.842481 23123 consensus_peers.cc:485] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5e1df757e4f40559ab9a3d5b9291550. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:00.842519 23319 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.842516 23449 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:00.843117 23829 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.843504 24166 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:00.847250 24128 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.848070 23449 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.848136 23319 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:00.852906 23045 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:01.094504 24239 ts_tablet_manager.cc:902] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:01.096525 24239 tablet_copy_client.cc:308] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:01.103852 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet 54227bfcbf9c4ad193075e5eec20a8ad from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:40127)
I20240503 02:22:01.104164 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet 54227bfcbf9c4ad193075e5eec20a8ad from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:40127: session id = 8f237b4e800e443fae82db47282e606f-54227bfcbf9c4ad193075e5eec20a8ad
I20240503 02:22:01.107275 23207 tablet_copy_source_session.cc:205] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:01.112893 24239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54227bfcbf9c4ad193075e5eec20a8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:01.127549 24239 tablet_copy_client.cc:783] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:01.128193 24239 tablet_copy_client.cc:647] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:01.130893 24239 tablet_copy_client.cc:523] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:01.135537 24239 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:01.147868 24239 log.cc:824] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:22:01.184902 24239 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:01.185680 24239 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:01.186184 24239 ts_tablet_manager.cc:1367] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.051s	user 0.051s	sys 0.000s
I20240503 02:22:01.190941 24239 raft_consensus.cc:348] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.191371 24239 raft_consensus.cc:729] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:01.191874 24239 consensus_queue.cc:260] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.193471 24239 ts_tablet_manager.cc:1398] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20240503 02:22:01.194535 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-54227bfcbf9c4ad193075e5eec20a8ad received from {username='slave'} at 127.20.218.68:40127
I20240503 02:22:01.194712 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session 8f237b4e800e443fae82db47282e606f-54227bfcbf9c4ad193075e5eec20a8ad on tablet 54227bfcbf9c4ad193075e5eec20a8ad with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:01.208825 24239 ts_tablet_manager.cc:902] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:01.210595 24239 tablet_copy_client.cc:308] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:01.211537 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet 100904c00d104bcca2956b21edf2c496 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:40127)
I20240503 02:22:01.211774 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet 100904c00d104bcca2956b21edf2c496 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:40127: session id = 8f237b4e800e443fae82db47282e606f-100904c00d104bcca2956b21edf2c496
I20240503 02:22:01.214109 23207 tablet_copy_source_session.cc:205] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:01.215577 24239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100904c00d104bcca2956b21edf2c496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:01.220554 24239 tablet_copy_client.cc:783] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:01.220976 24239 tablet_copy_client.cc:647] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:01.223098 24239 tablet_copy_client.cc:523] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:01.226190 24239 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:01.243602 24277 ts_tablet_manager.cc:902] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:01.245532 24277 tablet_copy_client.cc:308] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34401
I20240503 02:22:01.252499 23339 tablet_copy_service.cc:140] P b0c0b6bc47954db9ada48f0c09ee9959: Received BeginTabletCopySession request for tablet 5db050761e584724a0710ed437a0bb32 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:50321)
I20240503 02:22:01.252853 23339 tablet_copy_service.cc:161] P b0c0b6bc47954db9ada48f0c09ee9959: Beginning new tablet copy session on tablet 5db050761e584724a0710ed437a0bb32 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:50321: session id = 8f237b4e800e443fae82db47282e606f-5db050761e584724a0710ed437a0bb32
I20240503 02:22:01.255628 23339 tablet_copy_source_session.cc:205] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:01.257781 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db050761e584724a0710ed437a0bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:01.263315 24277 tablet_copy_client.cc:783] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:01.263608 24277 tablet_copy_client.cc:647] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:01.264225 24239 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:01.264751 24239 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:01.265105 24239 ts_tablet_manager.cc:1367] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.039s	user 0.035s	sys 0.000s
I20240503 02:22:01.266202 24239 raft_consensus.cc:348] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.266373 24277 tablet_copy_client.cc:523] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:01.266502 24239 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:01.266744 24239 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.267493 24239 ts_tablet_manager.cc:1398] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240503 02:22:01.268322 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-100904c00d104bcca2956b21edf2c496 received from {username='slave'} at 127.20.218.68:40127
I20240503 02:22:01.268469 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session 8f237b4e800e443fae82db47282e606f-100904c00d104bcca2956b21edf2c496 on tablet 100904c00d104bcca2956b21edf2c496 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:01.270184 24277 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:01.300695 24277 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:01.301162 24277 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:01.301476 24277 ts_tablet_manager.cc:1367] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.031s	user 0.027s	sys 0.005s
I20240503 02:22:01.302170 24277 raft_consensus.cc:348] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.302414 24277 raft_consensus.cc:729] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:01.302659 24277 consensus_queue.cc:260] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.303181 24277 ts_tablet_manager.cc:1398] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:01.304409 23339 tablet_copy_service.cc:342] P b0c0b6bc47954db9ada48f0c09ee9959: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-5db050761e584724a0710ed437a0bb32 received from {username='slave'} at 127.20.218.68:50321
I20240503 02:22:01.304652 23339 tablet_copy_service.cc:433] P b0c0b6bc47954db9ada48f0c09ee9959: ending tablet copy session 8f237b4e800e443fae82db47282e606f-5db050761e584724a0710ed437a0bb32 on tablet 5db050761e584724a0710ed437a0bb32 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:01.392549 24277 ts_tablet_manager.cc:902] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:22:01.393798 24277 tablet_copy_client.cc:308] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41461
I20240503 02:22:01.400125 24239 ts_tablet_manager.cc:902] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:01.400101 23471 tablet_copy_service.cc:140] P 7ec52a420ca6438991468d6d68818019: Received BeginTabletCopySession request for tablet bb8aafe8adda4229bbe493e76dc8aed4 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:59377)
I20240503 02:22:01.400360 23471 tablet_copy_service.cc:161] P 7ec52a420ca6438991468d6d68818019: Beginning new tablet copy session on tablet bb8aafe8adda4229bbe493e76dc8aed4 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:59377: session id = 8f237b4e800e443fae82db47282e606f-bb8aafe8adda4229bbe493e76dc8aed4
I20240503 02:22:01.407220 23471 tablet_copy_source_session.cc:205] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:01.410472 24277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8aafe8adda4229bbe493e76dc8aed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:01.422369 24277 tablet_copy_client.cc:783] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:01.422752 24277 tablet_copy_client.cc:647] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:01.423532 24239 tablet_copy_client.cc:308] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:01.424271 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet a5e1df757e4f40559ab9a3d5b9291550 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:40127)
I20240503 02:22:01.424497 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet a5e1df757e4f40559ab9a3d5b9291550 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:40127: session id = 8f237b4e800e443fae82db47282e606f-a5e1df757e4f40559ab9a3d5b9291550
I20240503 02:22:01.425426 24277 tablet_copy_client.cc:523] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:01.427911 23207 tablet_copy_source_session.cc:205] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:01.428221 24277 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:01.429842 24239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e1df757e4f40559ab9a3d5b9291550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:01.438233 24239 tablet_copy_client.cc:783] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:01.438592 24239 tablet_copy_client.cc:647] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:01.441200 24239 tablet_copy_client.cc:523] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:01.445371 24239 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:01.480566 24277 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:01.481154 24277 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:01.481487 24277 ts_tablet_manager.cc:1367] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.053s	user 0.044s	sys 0.011s
I20240503 02:22:01.482416 24277 raft_consensus.cc:348] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.482695 24277 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:01.482949 24277 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.483659 24277 ts_tablet_manager.cc:1398] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:22:01.484851 23471 tablet_copy_service.cc:342] P 7ec52a420ca6438991468d6d68818019: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-bb8aafe8adda4229bbe493e76dc8aed4 received from {username='slave'} at 127.20.218.68:59377
I20240503 02:22:01.485060 23471 tablet_copy_service.cc:433] P 7ec52a420ca6438991468d6d68818019: ending tablet copy session 8f237b4e800e443fae82db47282e606f-bb8aafe8adda4229bbe493e76dc8aed4 on tablet bb8aafe8adda4229bbe493e76dc8aed4 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:01.500635 24239 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:01.501427 24239 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:01.501993 24239 ts_tablet_manager.cc:1367] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.057s	user 0.042s	sys 0.016s
I20240503 02:22:01.503325 24239 raft_consensus.cc:348] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.504015 24239 raft_consensus.cc:729] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:01.504294 24239 consensus_queue.cc:260] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:01.504957 24239 ts_tablet_manager.cc:1398] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240503 02:22:01.505957 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-a5e1df757e4f40559ab9a3d5b9291550 received from {username='slave'} at 127.20.218.68:40127
I20240503 02:22:01.506155 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session 8f237b4e800e443fae82db47282e606f-a5e1df757e4f40559ab9a3d5b9291550 on tablet a5e1df757e4f40559ab9a3d5b9291550 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:01.546087 23977 raft_consensus.cc:1203] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:01.672952 23977 raft_consensus.cc:1203] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:01.745582 23977 raft_consensus.cc:1203] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:01.828595 23977 raft_consensus.cc:1203] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:02.017167 23977 raft_consensus.cc:1203] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:02.062206 24128 raft_consensus.cc:1050] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:02.063027 24128 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.065925 23450 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.065925 23319 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.066287 23977 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.066682 24374 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.067078 24318 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.067535 24128 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.071671 24374 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.072438 23977 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.077306 23043 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.079162 23319 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.082859 23450 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.088239 23186 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.090142 23977 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.090667 23450 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.090756 24318 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.091267 24128 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.093037 24318 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.094048 23977 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.094246 23450 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.097394 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54227bfcbf9c4ad193075e5eec20a8ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:02.098964 23043 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:02.103258 23297 tablet_service.cc:1514] Processing DeleteTablet for tablet 54227bfcbf9c4ad193075e5eec20a8ad with delete_type TABLET_DATA_TOMBSTONED (TS b0c0b6bc47954db9ada48f0c09ee9959 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37518
I20240503 02:22:02.104187 24382 tablet_replica.cc:330] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: stopping tablet replica
I20240503 02:22:02.104561 24382 raft_consensus.cc:2229] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:02.104997 24382 raft_consensus.cc:2258] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:02.106561 24382 ts_tablet_manager.cc:1875] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:02.111775 24382 ts_tablet_manager.cc:1888] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:22:02.111929 24382 log.cc:1196] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/54227bfcbf9c4ad193075e5eec20a8ad
I20240503 02:22:02.112824 23030 catalog_manager.cc:4830] TS b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): tablet 54227bfcbf9c4ad193075e5eec20a8ad (table rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]) successfully deleted
I20240503 02:22:02.131403 24334 raft_consensus.cc:1050] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:02.132225 24334 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.134797 23450 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.134837 23977 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.134936 23186 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.135414 24334 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.135787 24384 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.136129 24333 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.142993 24334 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.143812 23450 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.143944 23186 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.146806 23977 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.149320 23045 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.155012 23319 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.157653 23450 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.157620 23977 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.158254 24204 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.158668 24384 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.161101 24204 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.162474 23450 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.165388 23977 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.170679 23039 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.173817 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5db050761e584724a0710ed437a0bb32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:02.175062 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db050761e584724a0710ed437a0bb32 with delete_type TABLET_DATA_TOMBSTONED (TS dcbb067767bf4060bbac25f4f5fad61a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54744
I20240503 02:22:02.176116 24398 tablet_replica.cc:330] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: stopping tablet replica
I20240503 02:22:02.176607 24398 raft_consensus.cc:2229] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:02.177084 24398 raft_consensus.cc:2258] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:02.179217 24398 ts_tablet_manager.cc:1875] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:02.185218 24398 ts_tablet_manager.cc:1888] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:22:02.185467 24398 log.cc:1196] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/5db050761e584724a0710ed437a0bb32
I20240503 02:22:02.186728 23033 catalog_manager.cc:4830] TS dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): tablet 5db050761e584724a0710ed437a0bb32 (table rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]) successfully deleted
I20240503 02:22:02.224835 23319 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:02.227366 23450 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:02.227469 23186 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:22:02.227615 23255 consensus_peers.cc:485] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11e7201491ab41beb241006b9ce12624. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:02.228075 24384 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:02.228435 24334 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:02.239521 24334 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:02.240959 23186 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:02.242439 24318 raft_consensus.cc:1050] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:02.242434 24374 raft_consensus.cc:1050] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:02.243408 24374 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.243394 24318 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.245242 23450 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:02.247586 23043 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index -1 to 4, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:02.258255 23976 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.258328 23319 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.258503 23449 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.259029 24128 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.259640 23829 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.262430 23829 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.275234 24128 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.278664 23318 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.280190 23976 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.281029 24426 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.282310 23037 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:02.281675 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.291265 23186 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.291692 23450 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.292472 24426 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.295909 23319 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.290670 23449 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.297130 24374 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.301657 23977 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.303865 24318 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.304785 23448 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.306032 24318 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.309684 23829 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.312960 24128 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.313503 23319 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.316432 23448 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.317291 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5e1df757e4f40559ab9a3d5b9291550 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:02.317209 23037 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:02.319216 23976 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.321810 23297 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e1df757e4f40559ab9a3d5b9291550 with delete_type TABLET_DATA_TOMBSTONED (TS b0c0b6bc47954db9ada48f0c09ee9959 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37518
I20240503 02:22:02.322185 24382 tablet_replica.cc:330] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: stopping tablet replica
I20240503 02:22:02.322394 24382 raft_consensus.cc:2229] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:02.322719 24382 raft_consensus.cc:2258] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:02.324565 23043 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.326310 24382 ts_tablet_manager.cc:1875] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:02.325953 23448 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.330788 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.337015 24382 ts_tablet_manager.cc:1888] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:22:02.337188 24382 log.cc:1196] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/a5e1df757e4f40559ab9a3d5b9291550
I20240503 02:22:02.338137 23030 catalog_manager.cc:4830] TS b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): tablet a5e1df757e4f40559ab9a3d5b9291550 (table rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]) successfully deleted
I20240503 02:22:02.376914 23186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "100904c00d104bcca2956b21edf2c496"
dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51298
I20240503 02:22:02.377213 23186 raft_consensus.cc:595] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:02.416745 23186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "100904c00d104bcca2956b21edf2c496"
dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51302
I20240503 02:22:02.416996 23186 raft_consensus.cc:595] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:02.508949 24371 raft_consensus.cc:1050] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:02.509951 24371 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.512910 23319 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.513311 23976 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.513587 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:02.514228 24192 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.514637 24372 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.515154 24192 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:02.527748 24371 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.528779 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.529284 23976 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.532905 23319 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.533653 23037 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { replace: true } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.540329 23448 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.542464 23976 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.542505 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:02.543032 24478 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.543355 24371 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:02.545717 24192 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.546486 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.546604 23976 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:02.548664 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb8aafe8adda4229bbe493e76dc8aed4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:02.550751 23039 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 5 to 6, VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) evicted. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:02.554744 23297 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8aafe8adda4229bbe493e76dc8aed4 with delete_type TABLET_DATA_TOMBSTONED (TS b0c0b6bc47954db9ada48f0c09ee9959 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37518
I20240503 02:22:02.555145 24382 tablet_replica.cc:330] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: stopping tablet replica
I20240503 02:22:02.555392 24382 raft_consensus.cc:2229] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:02.555672 24382 raft_consensus.cc:2258] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:02.556560 24382 ts_tablet_manager.cc:1875] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:02.561396 24382 ts_tablet_manager.cc:1888] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:22:02.561532 24382 log.cc:1196] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb8aafe8adda4229bbe493e76dc8aed4
I20240503 02:22:02.562295 23030 catalog_manager.cc:4830] TS b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): tablet bb8aafe8adda4229bbe493e76dc8aed4 (table rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]) successfully deleted
I20240503 02:22:02.657207 23186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "100904c00d104bcca2956b21edf2c496"
dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51328
I20240503 02:22:02.657526 23186 raft_consensus.cc:595] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:02.687644 23186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "100904c00d104bcca2956b21edf2c496"
dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:51356
I20240503 02:22:02.687851 23186 raft_consensus.cc:595] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:02.770495 24517 ts_tablet_manager.cc:902] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:02.771850 24517 tablet_copy_client.cc:308] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34401
I20240503 02:22:02.772580 23339 tablet_copy_service.cc:140] P b0c0b6bc47954db9ada48f0c09ee9959: Received BeginTabletCopySession request for tablet 11e7201491ab41beb241006b9ce12624 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:50321)
I20240503 02:22:02.772744 23339 tablet_copy_service.cc:161] P b0c0b6bc47954db9ada48f0c09ee9959: Beginning new tablet copy session on tablet 11e7201491ab41beb241006b9ce12624 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:50321: session id = 8f237b4e800e443fae82db47282e606f-11e7201491ab41beb241006b9ce12624
I20240503 02:22:02.774842 23339 tablet_copy_source_session.cc:205] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:02.776151 24517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e7201491ab41beb241006b9ce12624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:02.780535 24517 tablet_copy_client.cc:783] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:02.780807 24517 tablet_copy_client.cc:647] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:02.782692 24517 tablet_copy_client.cc:523] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:02.785291 24517 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:02.811834 24430 raft_consensus.cc:979] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: : Instructing follower b0c0b6bc47954db9ada48f0c09ee9959 to start an election
I20240503 02:22:02.812161 24430 raft_consensus.cc:1067] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Signalling peer b0c0b6bc47954db9ada48f0c09ee9959 to start an election
I20240503 02:22:02.813254 23319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "100904c00d104bcca2956b21edf2c496"
dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
 from {username='slave'} at 127.20.218.65:36089
I20240503 02:22:02.813611 23319 raft_consensus.cc:482] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:22:02.813752 23319 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:02.816812 23319 raft_consensus.cc:504] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.817797 23319 leader_election.cc:288] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 2 election: Requested vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461), 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297)
I20240503 02:22:02.818181 23186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:22:02.818435 23186 raft_consensus.cc:3032] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:22:02.818557 23186 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:22:02.818559 23976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8f237b4e800e443fae82db47282e606f"
I20240503 02:22:02.818925 23976 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:02.818925 23186 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.819383 23186 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:02.821800 24517 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:02.822145 24517 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:02.822360 23448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100904c00d104bcca2956b21edf2c496" candidate_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:22:02.822472 23976 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 2.
I20240503 02:22:02.822587 23448 raft_consensus.cc:3037] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:02.822474 24517 ts_tablet_manager.cc:1367] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.037s	user 0.030s	sys 0.008s
I20240503 02:22:02.823557 23186 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 2.
I20240503 02:22:02.823742 24517 raft_consensus.cc:348] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:02.823993 24517 raft_consensus.cc:729] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:02.824158 23255 leader_election.cc:302] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8f237b4e800e443fae82db47282e606f, b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:22:02.824223 24517 consensus_queue.cc:260] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:02.824517 24334 raft_consensus.cc:2781] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:22:02.824756 24517 ts_tablet_manager.cc:1398] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:02.825088 23448 raft_consensus.cc:2454] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c0b6bc47954db9ada48f0c09ee9959 in term 2.
I20240503 02:22:02.825553 23339 tablet_copy_service.cc:342] P b0c0b6bc47954db9ada48f0c09ee9959: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-11e7201491ab41beb241006b9ce12624 received from {username='slave'} at 127.20.218.68:50321
I20240503 02:22:02.825708 23339 tablet_copy_service.cc:433] P b0c0b6bc47954db9ada48f0c09ee9959: ending tablet copy session 8f237b4e800e443fae82db47282e606f-11e7201491ab41beb241006b9ce12624 on tablet 11e7201491ab41beb241006b9ce12624 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:02.829651 24334 raft_consensus.cc:686] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Becoming Leader. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Running, Role: LEADER
I20240503 02:22:02.830024 24334 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:02.832525 23043 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: term changed from 1 to 2, leader changed from dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) to b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66). New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:03.274780 23976 raft_consensus.cc:1203] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240503 02:22:03.276942 23976 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:22:03.277499 24544 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:03.431577 23186 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:22:03.432205 24544 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:03.436631 23451 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:22:03.437661 24334 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:03.777947 24544 raft_consensus.cc:1050] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:03.778893 24544 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:03.781680 23976 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:03.781714 23451 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:03.781986 23186 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:03.782465 24334 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:03.782860 24523 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:03.783172 24544 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:03.791042 24523 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.791841 23451 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.792510 23976 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.798520 23039 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.800886 23186 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { replace: true } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.816283 23319 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:03.819096 23451 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:03.819794 24586 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:03.820331 23976 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:03.821702 24586 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:03.823251 24523 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.823879 23451 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.824527 23319 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:03.826367 23448 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:22:03.826820 24544 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:03.827252 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11e7201491ab41beb241006b9ce12624 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:03.828874 23977 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.829269 23043 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 5 to 6, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.830174 24544 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.830822 23451 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.833097 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 100904c00d104bcca2956b21edf2c496 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:03.834581 23976 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:22:03.835093 24544 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:03.836421 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e7201491ab41beb241006b9ce12624 with delete_type TABLET_DATA_TOMBSTONED (TS dcbb067767bf4060bbac25f4f5fad61a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54744
I20240503 02:22:03.837195 24591 tablet_replica.cc:330] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: stopping tablet replica
I20240503 02:22:03.837754 24591 raft_consensus.cc:2229] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:03.838136 24591 raft_consensus.cc:2258] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:03.837987 23045 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 5 to 7, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.838968 24591 ts_tablet_manager.cc:1875] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:03.839576 23976 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:03.844014 23167 tablet_service.cc:1514] Processing DeleteTablet for tablet 100904c00d104bcca2956b21edf2c496 with delete_type TABLET_DATA_TOMBSTONED (TS dcbb067767bf4060bbac25f4f5fad61a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54744
I20240503 02:22:03.845113 24591 ts_tablet_manager.cc:1888] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:22:03.845263 24591 log.cc:1196] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/11e7201491ab41beb241006b9ce12624
I20240503 02:22:03.846127 24591 tablet_replica.cc:330] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: stopping tablet replica
I20240503 02:22:03.846323 24591 raft_consensus.cc:2229] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:03.846289 23033 catalog_manager.cc:4830] TS dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): tablet 11e7201491ab41beb241006b9ce12624 (table rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]) successfully deleted
I20240503 02:22:03.846510 24591 raft_consensus.cc:2258] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:03.847294 24591 ts_tablet_manager.cc:1875] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:03.851446 24591 ts_tablet_manager.cc:1888] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:22:03.851560 24591 log.cc:1196] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-0/wal/wals/100904c00d104bcca2956b21edf2c496
I20240503 02:22:03.852280 23033 catalog_manager.cc:4830] TS dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): tablet 100904c00d104bcca2956b21edf2c496 (table rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]) successfully deleted
I20240503 02:22:04.141742 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:04.143708 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:04.147122 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:04.152282 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:04.338236 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:04.340142 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:04.344149 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:04.355773 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:04.527092 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:04.535756 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:04.537299 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:04.539497 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:05.169222 23045 ts_manager.cc:292] Set tserver state for 7ec52a420ca6438991468d6d68818019 to MAINTENANCE_MODE
I20240503 02:22:06.737520 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:06.737520 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:06.748910 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:06.758003 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:06.926985 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:06.927932 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:06.928668 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:06.936543 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:07.109427 23429 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:22:07.109560 23957 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:07.118985 23167 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:22:07.132069 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:07.189318 23186 consensus_queue.cc:237] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } }
I20240503 02:22:07.191654 23451 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:07.191968 23319 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:07.192237 24569 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:07.192657 24824 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:07.196122 24824 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:07.196751 23319 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:07.197103 23451 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:07.202066 23039 catalog_manager.cc:5484] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:07.210954 23186 consensus_queue.cc:237] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
W20240503 02:22:07.213538 23123 consensus_peers.cc:485] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50bb8de00f2046c3b4ff426bbc87b37f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:07.213773 23319 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:07.213907 23451 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:07.214537 24824 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:07.214923 24569 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:07.217653 24824 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:07.218586 23451 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:07.219091 23319 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:07.222898 23186 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:07.223179 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 50bb8de00f2046c3b4ff426bbc87b37f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:07.224414 23045 catalog_manager.cc:5484] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:07.227169 23976 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:07.227355 23448 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:07.227768 24569 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:07.228091 24827 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:07.231171 24824 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.231732 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.233132 23448 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.237054 23045 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:07.244015 23186 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:07.246879 23976 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:07.247174 23448 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:07.247442 24569 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:07.247807 24824 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:22:07.248149 23120 consensus_peers.cc:485] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a -> Peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): Couldn't send request to peer b0c0b6bc47954db9ada48f0c09ee9959. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:07.250584 24824 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:07.251358 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:07.254626 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a5e1df757e4f40559ab9a3d5b9291550 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:07.254797 23448 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:07.257772 23045 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:07.258433 23319 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:07.260972 23448 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:07.261060 23976 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:07.261749 24787 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:07.262168 24586 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:07.265759 24586 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.266419 23976 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.271337 23043 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.271524 23448 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:07.278457 23319 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:07.280678 23448 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:07.280825 23976 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 02:22:07.281283 23255 consensus_peers.cc:485] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): Couldn't send request to peer dcbb067767bf4060bbac25f4f5fad61a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:07.281358 24586 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:07.282534 24586 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:07.285087 24854 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:07.285887 23448 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:07.287009 23976 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:07.291746 23039 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:07.292256 23030 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5db050761e584724a0710ed437a0bb32 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:07.624053 24880 ts_tablet_manager.cc:902] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:07.625305 24880 tablet_copy_client.cc:308] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:07.626063 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet 50bb8de00f2046c3b4ff426bbc87b37f from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:40127)
I20240503 02:22:07.626227 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet 50bb8de00f2046c3b4ff426bbc87b37f from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:40127: session id = 8f237b4e800e443fae82db47282e606f-50bb8de00f2046c3b4ff426bbc87b37f
I20240503 02:22:07.628167 23207 tablet_copy_source_session.cc:205] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:07.629469 24880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50bb8de00f2046c3b4ff426bbc87b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:07.633934 24880 tablet_copy_client.cc:783] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:07.634207 24880 tablet_copy_client.cc:647] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:07.636031 24880 tablet_copy_client.cc:523] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:07.638649 24880 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:07.668489 24880 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:07.668900 24880 tablet_bootstrap.cc:492] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:07.669131 24880 ts_tablet_manager.cc:1367] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.031s	user 0.023s	sys 0.004s
I20240503 02:22:07.669819 24880 raft_consensus.cc:348] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:07.670034 24880 raft_consensus.cc:729] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:07.670199 24880 consensus_queue.cc:260] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:07.670687 24880 ts_tablet_manager.cc:1398] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.003s
I20240503 02:22:07.671665 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-50bb8de00f2046c3b4ff426bbc87b37f received from {username='slave'} at 127.20.218.68:40127
I20240503 02:22:07.671792 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session 8f237b4e800e443fae82db47282e606f-50bb8de00f2046c3b4ff426bbc87b37f on tablet 50bb8de00f2046c3b4ff426bbc87b37f with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:07.794600 24901 ts_tablet_manager.cc:902] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Initiating tablet copy from peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:07.796172 24901 tablet_copy_client.cc:308] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34401
I20240503 02:22:07.797086 23339 tablet_copy_service.cc:140] P b0c0b6bc47954db9ada48f0c09ee9959: Received BeginTabletCopySession request for tablet 5db050761e584724a0710ed437a0bb32 from peer dcbb067767bf4060bbac25f4f5fad61a ({username='slave'} at 127.20.218.65:36089)
I20240503 02:22:07.797246 23339 tablet_copy_service.cc:161] P b0c0b6bc47954db9ada48f0c09ee9959: Beginning new tablet copy session on tablet 5db050761e584724a0710ed437a0bb32 from peer dcbb067767bf4060bbac25f4f5fad61a at {username='slave'} at 127.20.218.65:36089: session id = dcbb067767bf4060bbac25f4f5fad61a-5db050761e584724a0710ed437a0bb32
I20240503 02:22:07.799371 23339 tablet_copy_source_session.cc:205] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:07.800918 24901 ts_tablet_manager.cc:1875] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:07.805627 24901 ts_tablet_manager.cc:1888] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:22:07.805958 24901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db050761e584724a0710ed437a0bb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:07.808429 24901 tablet_copy_client.cc:783] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:07.808801 24901 tablet_copy_client.cc:647] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:07.810853 24901 tablet_copy_client.cc:523] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:07.814348 24901 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:22:07.834945 24907 ts_tablet_manager.cc:902] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:07.836982 24907 tablet_copy_client.cc:308] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:07.838052 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet a5e1df757e4f40559ab9a3d5b9291550 from peer b0c0b6bc47954db9ada48f0c09ee9959 ({username='slave'} at 127.20.218.66:55997)
I20240503 02:22:07.838243 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet a5e1df757e4f40559ab9a3d5b9291550 from peer b0c0b6bc47954db9ada48f0c09ee9959 at {username='slave'} at 127.20.218.66:55997: session id = b0c0b6bc47954db9ada48f0c09ee9959-a5e1df757e4f40559ab9a3d5b9291550
I20240503 02:22:07.841549 23207 tablet_copy_source_session.cc:205] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:07.843062 24907 ts_tablet_manager.cc:1875] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:07.847903 24907 ts_tablet_manager.cc:1888] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:22:07.848263 24907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e1df757e4f40559ab9a3d5b9291550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:07.851048 24907 tablet_copy_client.cc:783] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:07.851369 24907 tablet_copy_client.cc:647] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:07.852772 24901 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:07.853236 24901 tablet_bootstrap.cc:492] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap complete.
I20240503 02:22:07.853571 24907 tablet_copy_client.cc:523] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:07.853566 24901 ts_tablet_manager.cc:1367] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.039s	user 0.040s	sys 0.000s
I20240503 02:22:07.854482 24901 raft_consensus.cc:348] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:07.854787 24901 raft_consensus.cc:729] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: LEARNER
I20240503 02:22:07.855023 24901 consensus_queue.cc:260] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:07.855652 24901 ts_tablet_manager.cc:1398] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:07.856674 23339 tablet_copy_service.cc:342] P b0c0b6bc47954db9ada48f0c09ee9959: Request end of tablet copy session dcbb067767bf4060bbac25f4f5fad61a-5db050761e584724a0710ed437a0bb32 received from {username='slave'} at 127.20.218.65:36089
I20240503 02:22:07.856837 23339 tablet_copy_service.cc:433] P b0c0b6bc47954db9ada48f0c09ee9959: ending tablet copy session dcbb067767bf4060bbac25f4f5fad61a-5db050761e584724a0710ed437a0bb32 on tablet 5db050761e584724a0710ed437a0bb32 with peer dcbb067767bf4060bbac25f4f5fad61a
I20240503 02:22:07.858651 24907 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:22:07.893985 24907 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:07.894447 24907 tablet_bootstrap.cc:492] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap complete.
I20240503 02:22:07.894727 24907 ts_tablet_manager.cc:1367] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.003s
I20240503 02:22:07.895434 24907 raft_consensus.cc:348] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:07.895676 24907 raft_consensus.cc:729] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: LEARNER
I20240503 02:22:07.895884 24907 consensus_queue.cc:260] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:07.896412 24907 ts_tablet_manager.cc:1398] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:07.897413 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-a5e1df757e4f40559ab9a3d5b9291550 received from {username='slave'} at 127.20.218.66:55997
I20240503 02:22:07.897573 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-a5e1df757e4f40559ab9a3d5b9291550 on tablet a5e1df757e4f40559ab9a3d5b9291550 with peer b0c0b6bc47954db9ada48f0c09ee9959
I20240503 02:22:08.013100 23976 raft_consensus.cc:1203] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:22:08.278075 23186 raft_consensus.cc:1203] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:22:08.278570 23319 raft_consensus.cc:1203] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:22:08.548964 24928 raft_consensus.cc:1050] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:08.550091 24928 consensus_queue.cc:237] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:08.553174 23976 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:08.553346 23448 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:08.553309 23319 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:08.553993 24928 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:08.554342 24963 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:08.554674 24569 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:08.567509 24928 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.568296 23319 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.569181 23448 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.573997 23045 catalog_manager.cc:5484] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.575284 23976 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.586989 23186 consensus_queue.cc:237] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:08.588968 23976 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:08.589036 23319 raft_consensus.cc:1261] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:08.589639 24963 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:08.589987 24928 consensus_queue.cc:1035] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:08.592286 24963 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.593080 23319 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.595283 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50bb8de00f2046c3b4ff426bbc87b37f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:08.596560 23976 raft_consensus.cc:2932] T 50bb8de00f2046c3b4ff426bbc87b37f P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:08.599711 23039 catalog_manager.cc:5484] T 50bb8de00f2046c3b4ff426bbc87b37f P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:08.605482 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 50bb8de00f2046c3b4ff426bbc87b37f with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:08.606518 24972 tablet_replica.cc:330] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:08.606909 24972 raft_consensus.cc:2229] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:08.607280 24972 raft_consensus.cc:2258] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:08.608853 24972 ts_tablet_manager.cc:1875] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:08.614279 24972 ts_tablet_manager.cc:1888] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:22:08.614444 24972 log.cc:1196] T 50bb8de00f2046c3b4ff426bbc87b37f P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/50bb8de00f2046c3b4ff426bbc87b37f
I20240503 02:22:08.615326 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 50bb8de00f2046c3b4ff426bbc87b37f (table rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]) successfully deleted
I20240503 02:22:08.733397 24946 raft_consensus.cc:1050] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959: attempting to promote NON_VOTER dcbb067767bf4060bbac25f4f5fad61a to VOTER
I20240503 02:22:08.734158 24946 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:08.736775 23976 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.736843 23186 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.736960 23448 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.737505 24989 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.737969 24854 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.738253 24946 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.746393 24946 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.747170 23976 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.747269 23448 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.751763 23043 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:08.753405 23186 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.756640 23319 consensus_queue.cc:237] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:08.759076 23976 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:08.759177 23187 raft_consensus.cc:1261] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:08.759860 24989 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:08.760322 24946 consensus_queue.cc:1035] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:08.762861 24854 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.766378 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5db050761e584724a0710ed437a0bb32 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:08.766119 23186 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.766871 23976 raft_consensus.cc:2932] T 5db050761e584724a0710ed437a0bb32 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:08.767864 23039 catalog_manager.cc:5484] T 5db050761e584724a0710ed437a0bb32 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:08.771584 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db050761e584724a0710ed437a0bb32 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:08.771960 24972 tablet_replica.cc:330] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:08.772192 24972 raft_consensus.cc:2229] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:08.772711 24972 raft_consensus.cc:2258] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:08.774468 24972 ts_tablet_manager.cc:1875] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:08.779412 24972 ts_tablet_manager.cc:1888] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:22:08.779536 24972 log.cc:1196] T 5db050761e584724a0710ed437a0bb32 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/5db050761e584724a0710ed437a0bb32
I20240503 02:22:08.780243 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 5db050761e584724a0710ed437a0bb32 (table rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]) successfully deleted
I20240503 02:22:08.867487 24928 raft_consensus.cc:1050] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 to VOTER
I20240503 02:22:08.868196 24928 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:08.870571 23319 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.870572 23448 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.870594 23976 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:08.871181 24569 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.871515 24928 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.871740 24963 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:08.878875 24928 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.879892 23319 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.882516 23448 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.882483 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.884820 23037 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.890514 23186 consensus_queue.cc:237] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:08.892418 23319 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:08.892407 23976 raft_consensus.cc:1261] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:08.893090 24929 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:08.893891 24929 consensus_queue.cc:1035] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:08.895867 24963 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.896692 23976 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.896737 23319 raft_consensus.cc:2932] T a5e1df757e4f40559ab9a3d5b9291550 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.901016 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a5e1df757e4f40559ab9a3d5b9291550 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:08.901926 23037 catalog_manager.cc:5484] T a5e1df757e4f40559ab9a3d5b9291550 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:08.905725 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e1df757e4f40559ab9a3d5b9291550 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:08.906101 24972 tablet_replica.cc:330] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:08.906292 24972 raft_consensus.cc:2229] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:08.906664 24972 raft_consensus.cc:2258] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:08.907568 24972 ts_tablet_manager.cc:1875] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:08.912396 24972 ts_tablet_manager.cc:1888] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:22:08.912536 24972 log.cc:1196] T a5e1df757e4f40559ab9a3d5b9291550 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/a5e1df757e4f40559ab9a3d5b9291550
I20240503 02:22:08.913257 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet a5e1df757e4f40559ab9a3d5b9291550 (table rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]) successfully deleted
I20240503 02:22:08.928257 23186 consensus_queue.cc:237] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } }
I20240503 02:22:08.930370 23448 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:08.930371 23319 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:08.930905 24928 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:08.931183 24963 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:08.933771 24569 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:08.934461 23448 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:08.934504 23319 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:08.938804 23043 catalog_manager.cc:5484] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 02:22:08.944041 23186 consensus_queue.cc:237] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:08.946271 23448 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:08.946271 23319 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:22:08.946502 23123 consensus_peers.cc:485] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61e4ce4f9e734b93b86d351eb411050c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:08.946888 24569 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:08.947252 24929 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:08.949959 24928 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:08.951220 23448 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:08.953541 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 61e4ce4f9e734b93b86d351eb411050c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:08.954362 23319 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:08.954983 23037 catalog_manager.cc:5484] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:09.204123 23448 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:09.206379 23977 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:09.206933 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:09.206946 24714 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:09.207505 25027 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:09.209968 24714 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.210575 23977 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.211330 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.217315 23043 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.222561 23448 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
W20240503 02:22:09.225005 23384 consensus_peers.cc:485] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 -> Peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): Couldn't send request to peer b0c0b6bc47954db9ada48f0c09ee9959. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:09.225231 23977 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:09.225517 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:09.225970 25030 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:09.226449 24714 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:09.229844 25027 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:09.230597 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:09.231051 23977 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:09.233557 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bb8aafe8adda4229bbe493e76dc8aed4 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:09.235476 23037 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:09.240324 23319 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:09.242976 23977 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:09.243088 23448 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:09.243630 24854 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:09.244019 24989 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:09.247171 24945 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.247848 23448 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.251542 23977 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.256026 23039 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:09.261845 23319 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:09.264283 23977 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:09.264720 23448 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:09.265221 24854 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:22:09.266880 23255 consensus_peers.cc:485] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): Couldn't send request to peer dcbb067767bf4060bbac25f4f5fad61a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:09.267306 24854 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:09.269398 24854 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:09.269891 23977 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:09.275036 23030 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 11e7201491ab41beb241006b9ce12624 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:09.274864 23037 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:09.274864 23448 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:09.426179 25054 ts_tablet_manager.cc:902] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:09.427547 25054 tablet_copy_client.cc:308] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:09.428295 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet 61e4ce4f9e734b93b86d351eb411050c from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:40127)
I20240503 02:22:09.428478 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet 61e4ce4f9e734b93b86d351eb411050c from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:40127: session id = 8f237b4e800e443fae82db47282e606f-61e4ce4f9e734b93b86d351eb411050c
I20240503 02:22:09.430465 23207 tablet_copy_source_session.cc:205] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:09.431828 25054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61e4ce4f9e734b93b86d351eb411050c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:09.436496 25054 tablet_copy_client.cc:783] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:09.436872 25054 tablet_copy_client.cc:647] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:09.438970 25054 tablet_copy_client.cc:523] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:09.441511 25054 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:09.472683 25054 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:09.473273 25054 tablet_bootstrap.cc:492] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:09.473594 25054 ts_tablet_manager.cc:1367] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.032s	user 0.023s	sys 0.008s
I20240503 02:22:09.474349 25054 raft_consensus.cc:348] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:09.474560 25054 raft_consensus.cc:729] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:09.474777 25054 consensus_queue.cc:260] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:09.475266 25054 ts_tablet_manager.cc:1398] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:09.476277 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-61e4ce4f9e734b93b86d351eb411050c received from {username='slave'} at 127.20.218.68:40127
I20240503 02:22:09.476394 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session 8f237b4e800e443fae82db47282e606f-61e4ce4f9e734b93b86d351eb411050c on tablet 61e4ce4f9e734b93b86d351eb411050c with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:09.691661 25075 ts_tablet_manager.cc:902] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Initiating tablet copy from peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:09.692905 25075 tablet_copy_client.cc:308] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34401
I20240503 02:22:09.693718 23339 tablet_copy_service.cc:140] P b0c0b6bc47954db9ada48f0c09ee9959: Received BeginTabletCopySession request for tablet 11e7201491ab41beb241006b9ce12624 from peer dcbb067767bf4060bbac25f4f5fad61a ({username='slave'} at 127.20.218.65:36089)
I20240503 02:22:09.693888 23339 tablet_copy_service.cc:161] P b0c0b6bc47954db9ada48f0c09ee9959: Beginning new tablet copy session on tablet 11e7201491ab41beb241006b9ce12624 from peer dcbb067767bf4060bbac25f4f5fad61a at {username='slave'} at 127.20.218.65:36089: session id = dcbb067767bf4060bbac25f4f5fad61a-11e7201491ab41beb241006b9ce12624
I20240503 02:22:09.695958 23339 tablet_copy_source_session.cc:205] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:09.697105 25075 ts_tablet_manager.cc:1875] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:09.701598 25075 ts_tablet_manager.cc:1888] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:22:09.701959 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e7201491ab41beb241006b9ce12624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:09.704726 25075 tablet_copy_client.cc:783] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:09.705076 25075 tablet_copy_client.cc:647] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:09.706822 25075 tablet_copy_client.cc:523] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:09.709518 25075 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:22:09.749636 25075 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:09.750065 25075 tablet_bootstrap.cc:492] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap complete.
I20240503 02:22:09.750325 25075 ts_tablet_manager.cc:1367] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.041s	user 0.028s	sys 0.011s
I20240503 02:22:09.751019 25075 raft_consensus.cc:348] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:09.751257 25075 raft_consensus.cc:729] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: LEARNER
I20240503 02:22:09.751477 25075 consensus_queue.cc:260] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:09.752072 25075 ts_tablet_manager.cc:1398] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240503 02:22:09.753013 23339 tablet_copy_service.cc:342] P b0c0b6bc47954db9ada48f0c09ee9959: Request end of tablet copy session dcbb067767bf4060bbac25f4f5fad61a-11e7201491ab41beb241006b9ce12624 received from {username='slave'} at 127.20.218.65:36089
I20240503 02:22:09.753140 23339 tablet_copy_service.cc:433] P b0c0b6bc47954db9ada48f0c09ee9959: ending tablet copy session dcbb067767bf4060bbac25f4f5fad61a-11e7201491ab41beb241006b9ce12624 on tablet 11e7201491ab41beb241006b9ce12624 with peer dcbb067767bf4060bbac25f4f5fad61a
I20240503 02:22:09.769819 25096 ts_tablet_manager.cc:902] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Initiating tablet copy from peer 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:22:09.771394 25096 tablet_copy_client.cc:308] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41461
I20240503 02:22:09.772102 23471 tablet_copy_service.cc:140] P 7ec52a420ca6438991468d6d68818019: Received BeginTabletCopySession request for tablet bb8aafe8adda4229bbe493e76dc8aed4 from peer b0c0b6bc47954db9ada48f0c09ee9959 ({username='slave'} at 127.20.218.66:38821)
I20240503 02:22:09.772264 23471 tablet_copy_service.cc:161] P 7ec52a420ca6438991468d6d68818019: Beginning new tablet copy session on tablet bb8aafe8adda4229bbe493e76dc8aed4 from peer b0c0b6bc47954db9ada48f0c09ee9959 at {username='slave'} at 127.20.218.66:38821: session id = b0c0b6bc47954db9ada48f0c09ee9959-bb8aafe8adda4229bbe493e76dc8aed4
I20240503 02:22:09.774777 23471 tablet_copy_source_session.cc:205] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:09.775719 25096 ts_tablet_manager.cc:1875] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:09.780292 25096 ts_tablet_manager.cc:1888] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:22:09.780568 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8aafe8adda4229bbe493e76dc8aed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:09.782850 25096 tablet_copy_client.cc:783] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:09.783118 25096 tablet_copy_client.cc:647] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:09.784755 25096 tablet_copy_client.cc:523] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:09.787426 25096 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:22:09.820531 25096 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:09.820971 25096 tablet_bootstrap.cc:492] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap complete.
I20240503 02:22:09.821245 25096 ts_tablet_manager.cc:1367] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.034s	user 0.027s	sys 0.005s
I20240503 02:22:09.821952 25096 raft_consensus.cc:348] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:09.822175 25096 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: LEARNER
I20240503 02:22:09.822362 25096 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:09.822831 25096 ts_tablet_manager.cc:1398] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240503 02:22:09.823654 23471 tablet_copy_service.cc:342] P 7ec52a420ca6438991468d6d68818019: Request end of tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-bb8aafe8adda4229bbe493e76dc8aed4 received from {username='slave'} at 127.20.218.66:38821
I20240503 02:22:09.823782 23471 tablet_copy_service.cc:433] P 7ec52a420ca6438991468d6d68818019: ending tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-bb8aafe8adda4229bbe493e76dc8aed4 on tablet bb8aafe8adda4229bbe493e76dc8aed4 with peer b0c0b6bc47954db9ada48f0c09ee9959
I20240503 02:22:09.986012 23977 raft_consensus.cc:1203] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:22:10.225662 23186 raft_consensus.cc:1203] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:22:10.235656 23319 raft_consensus.cc:1203] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:22:10.403892 25110 raft_consensus.cc:1050] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:10.404642 25110 consensus_queue.cc:237] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:10.407153 23448 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:10.407153 23977 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:10.407330 23319 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:10.407835 25110 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:10.408176 24929 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:10.408457 25109 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:10.411741 25109 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.412410 23319 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.413067 23977 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.417444 23039 catalog_manager.cc:5484] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.417513 23448 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.425226 23186 consensus_queue.cc:237] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:10.427204 23319 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:10.427219 23977 raft_consensus.cc:1261] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:10.427856 25110 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:10.428336 25140 consensus_queue.cc:1035] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:10.430368 25110 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.431162 23977 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.431464 23319 raft_consensus.cc:2932] T 61e4ce4f9e734b93b86d351eb411050c P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.433251 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61e4ce4f9e734b93b86d351eb411050c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:10.434629 23045 catalog_manager.cc:5484] T 61e4ce4f9e734b93b86d351eb411050c P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 6 to 7, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:10.439571 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 61e4ce4f9e734b93b86d351eb411050c with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:10.440642 25148 tablet_replica.cc:330] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:10.440819 25148 raft_consensus.cc:2229] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:10.441115 25148 raft_consensus.cc:2258] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:10.441932 25148 ts_tablet_manager.cc:1875] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:10.446296 25148 ts_tablet_manager.cc:1888] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:22:10.446424 25148 log.cc:1196] T 61e4ce4f9e734b93b86d351eb411050c P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/61e4ce4f9e734b93b86d351eb411050c
I20240503 02:22:10.447222 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 61e4ce4f9e734b93b86d351eb411050c (table rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]) successfully deleted
I20240503 02:22:10.628168 24854 raft_consensus.cc:1050] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959: attempting to promote NON_VOTER dcbb067767bf4060bbac25f4f5fad61a to VOTER
I20240503 02:22:10.629014 24854 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:10.631466 23186 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEARNER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.631553 23977 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.631553 23448 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.632052 24854 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.632403 25165 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.632753 25125 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.635942 24854 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.636572 23977 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.641420 23045 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.642611 23186 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.643481 23448 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
W20240503 02:22:10.647907 25165 consensus_queue.cc:1159] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7ec52a420ca6438991468d6d68818019" responder_term: 1 status { last_received { term: 1 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 9 } }
I20240503 02:22:10.648329 23319 consensus_queue.cc:237] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:10.650197 23186 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:10.650763 25125 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:10.651410 23977 raft_consensus.cc:1261] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:10.652119 25125 consensus_queue.cc:1035] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:10.652944 25165 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.653650 23186 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.655972 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11e7201491ab41beb241006b9ce12624 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:10.657649 23045 catalog_manager.cc:5484] T 11e7201491ab41beb241006b9ce12624 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:10.659272 23977 raft_consensus.cc:2932] T 11e7201491ab41beb241006b9ce12624 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:10.661782 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e7201491ab41beb241006b9ce12624 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:10.662179 25148 tablet_replica.cc:330] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:10.662384 25148 raft_consensus.cc:2229] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:10.662859 25148 raft_consensus.cc:2258] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:10.664304 25148 ts_tablet_manager.cc:1875] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:10.668887 25148 ts_tablet_manager.cc:1888] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:22:10.669010 25148 log.cc:1196] T 11e7201491ab41beb241006b9ce12624 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/11e7201491ab41beb241006b9ce12624
I20240503 02:22:10.669660 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 11e7201491ab41beb241006b9ce12624 (table rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]) successfully deleted
I20240503 02:22:10.697443 23448 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } }
I20240503 02:22:10.699510 23319 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:10.699671 23186 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:22:10.700122 25128 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:10.700451 25129 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:22:10.703249 24714 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:10.703902 23319 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:10.707265 23186 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } }
I20240503 02:22:10.708084 23037 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240503 02:22:10.714394 23448 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
W20240503 02:22:10.717502 23387 consensus_peers.cc:485] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 -> Peer 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297): Couldn't send request to peer 8f237b4e800e443fae82db47282e606f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beb46d2ddb004a6d8498000a326f2d29. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:10.718196 23186 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:10.719435 23319 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:22:10.719597 24714 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:10.719998 25129 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:22:10.722185 25129 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:10.723330 23319 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:10.724443 23186 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } } }
I20240503 02:22:10.728202 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet beb46d2ddb004a6d8498000a326f2d29 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:10.728825 23037 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 4 to 5, NON_VOTER 8f237b4e800e443fae82db47282e606f (127.20.218.68) added. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:10.731320 25128 raft_consensus.cc:1050] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: attempting to promote NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 to VOTER
I20240503 02:22:10.732146 25128 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:10.737501 23186 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.738601 25128 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.738763 23977 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.739701 23318 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:10.739645 25128 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.740957 24714 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:10.744414 24714 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:10.745740 23318 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:10.746271 23977 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:10.749619 23186 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:10.751634 23045 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:10.776011 23448 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4"
dest_uuid: "7ec52a420ca6438991468d6d68818019"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35300
I20240503 02:22:10.776283 23448 raft_consensus.cc:595] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:10.991622 23318 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:10.993777 23448 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:22:10.993803 23977 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:22:10.994346 25165 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:10.994659 25125 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:10.997545 24854 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:10.998302 23977 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:11.000918 23448 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:11.003322 23039 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 7 to 8. New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:11.010289 23318 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
W20240503 02:22:11.013636 23255 consensus_peers.cc:485] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): Couldn't send request to peer dcbb067767bf4060bbac25f4f5fad61a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:11.014207 23977 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:22:11.014940 23448 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:22:11.015180 24854 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:11.016398 24854 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:11.018172 25125 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:11.019232 23977 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:11.021268 23030 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 100904c00d104bcca2956b21edf2c496 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:11.023078 23037 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 8 to 9, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:11.025449 23448 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } } }
I20240503 02:22:11.062726 23448 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4"
dest_uuid: "7ec52a420ca6438991468d6d68818019"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35330
I20240503 02:22:11.062952 23448 raft_consensus.cc:595] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:11.174314 25128 raft_consensus.cc:979] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: : Instructing follower dcbb067767bf4060bbac25f4f5fad61a to start an election
I20240503 02:22:11.174604 25128 raft_consensus.cc:1067] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Signalling peer dcbb067767bf4060bbac25f4f5fad61a to start an election
I20240503 02:22:11.175710 23186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4"
dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
 from {username='slave'} at 127.20.218.67:48233
I20240503 02:22:11.176043 23186 raft_consensus.cc:482] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:22:11.176191 23186 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:11.179908 23186 raft_consensus.cc:504] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:11.181440 23186 leader_election.cc:288] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 2 election: Requested vote from peers 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461), 8f237b4e800e443fae82db47282e606f (127.20.218.68:41297), b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:11.181852 23448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:22:11.181847 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8f237b4e800e443fae82db47282e606f"
I20240503 02:22:11.182068 23977 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:11.182125 23448 raft_consensus.cc:3032] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:22:11.182241 23448 raft_consensus.cc:729] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Running, Role: LEADER
I20240503 02:22:11.182444 23318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8aafe8adda4229bbe493e76dc8aed4" candidate_uuid: "dcbb067767bf4060bbac25f4f5fad61a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:22:11.182703 23318 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:11.182591 23448 consensus_queue.cc:260] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:11.183084 23448 raft_consensus.cc:3037] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:11.185954 23977 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 2.
I20240503 02:22:11.186131 23448 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 2.
I20240503 02:22:11.186175 23318 raft_consensus.cc:2454] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcbb067767bf4060bbac25f4f5fad61a in term 2.
I20240503 02:22:11.186801 23123 leader_election.cc:302] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7ec52a420ca6438991468d6d68818019, 8f237b4e800e443fae82db47282e606f, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:22:11.187141 25140 raft_consensus.cc:2781] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:22:11.187367 25140 raft_consensus.cc:686] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEADER]: Becoming Leader. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Running, Role: LEADER
I20240503 02:22:11.187752 25140 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:11.191597 23039 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: term changed from 1 to 2, leader changed from 7ec52a420ca6438991468d6d68818019 (127.20.218.67) to dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65). New cstate: current_term: 2 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:22:11.308727 25244 ts_tablet_manager.cc:902] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Initiating tablet copy from peer 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:22:11.317085 25244 tablet_copy_client.cc:308] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.67:41461
I20240503 02:22:11.317895 23471 tablet_copy_service.cc:140] P 7ec52a420ca6438991468d6d68818019: Received BeginTabletCopySession request for tablet beb46d2ddb004a6d8498000a326f2d29 from peer 8f237b4e800e443fae82db47282e606f ({username='slave'} at 127.20.218.68:59377)
I20240503 02:22:11.318049 23471 tablet_copy_service.cc:161] P 7ec52a420ca6438991468d6d68818019: Beginning new tablet copy session on tablet beb46d2ddb004a6d8498000a326f2d29 from peer 8f237b4e800e443fae82db47282e606f at {username='slave'} at 127.20.218.68:59377: session id = 8f237b4e800e443fae82db47282e606f-beb46d2ddb004a6d8498000a326f2d29
I20240503 02:22:11.320281 23471 tablet_copy_source_session.cc:205] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:11.321838 25244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb46d2ddb004a6d8498000a326f2d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:11.330619 25244 tablet_copy_client.cc:783] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:11.331017 25244 tablet_copy_client.cc:647] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:11.333209 25244 tablet_copy_client.cc:523] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:11.336210 25244 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Bootstrap starting.
I20240503 02:22:11.368034 25244 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:11.368451 25244 tablet_bootstrap.cc:492] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Bootstrap complete.
I20240503 02:22:11.368707 25244 ts_tablet_manager.cc:1367] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Time spent bootstrapping tablet: real 0.033s	user 0.032s	sys 0.000s
I20240503 02:22:11.369411 25244 raft_consensus.cc:348] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:11.369655 25244 raft_consensus.cc:729] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Initialized, Role: LEARNER
I20240503 02:22:11.369830 25244 consensus_queue.cc:260] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: NON_VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: true } }
I20240503 02:22:11.370359 25244 ts_tablet_manager.cc:1398] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240503 02:22:11.371208 23471 tablet_copy_service.cc:342] P 7ec52a420ca6438991468d6d68818019: Request end of tablet copy session 8f237b4e800e443fae82db47282e606f-beb46d2ddb004a6d8498000a326f2d29 received from {username='slave'} at 127.20.218.68:59377
I20240503 02:22:11.371371 23471 tablet_copy_service.cc:433] P 7ec52a420ca6438991468d6d68818019: ending tablet copy session 8f237b4e800e443fae82db47282e606f-beb46d2ddb004a6d8498000a326f2d29 on tablet beb46d2ddb004a6d8498000a326f2d29 with peer 8f237b4e800e443fae82db47282e606f
I20240503 02:22:11.465874 25252 ts_tablet_manager.cc:902] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Initiating tablet copy from peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401)
I20240503 02:22:11.467242 25252 tablet_copy_client.cc:308] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.66:34401
I20240503 02:22:11.468075 23339 tablet_copy_service.cc:140] P b0c0b6bc47954db9ada48f0c09ee9959: Received BeginTabletCopySession request for tablet 100904c00d104bcca2956b21edf2c496 from peer dcbb067767bf4060bbac25f4f5fad61a ({username='slave'} at 127.20.218.65:36089)
I20240503 02:22:11.468243 23339 tablet_copy_service.cc:161] P b0c0b6bc47954db9ada48f0c09ee9959: Beginning new tablet copy session on tablet 100904c00d104bcca2956b21edf2c496 from peer dcbb067767bf4060bbac25f4f5fad61a at {username='slave'} at 127.20.218.65:36089: session id = dcbb067767bf4060bbac25f4f5fad61a-100904c00d104bcca2956b21edf2c496
I20240503 02:22:11.470224 23339 tablet_copy_source_session.cc:205] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:11.471251 25252 ts_tablet_manager.cc:1875] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:11.475695 25252 ts_tablet_manager.cc:1888] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240503 02:22:11.475982 25252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100904c00d104bcca2956b21edf2c496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:11.478216 25252 tablet_copy_client.cc:783] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:11.478449 25252 tablet_copy_client.cc:647] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:11.480382 25252 tablet_copy_client.cc:523] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:11.483109 25252 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap starting.
I20240503 02:22:11.517045 25252 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:11.517515 25252 tablet_bootstrap.cc:492] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Bootstrap complete.
I20240503 02:22:11.517787 25252 ts_tablet_manager.cc:1367] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Time spent bootstrapping tablet: real 0.035s	user 0.032s	sys 0.004s
I20240503 02:22:11.518430 25252 raft_consensus.cc:348] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:11.518734 25252 raft_consensus.cc:729] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dcbb067767bf4060bbac25f4f5fad61a, State: Initialized, Role: LEARNER
I20240503 02:22:11.518980 25252 consensus_queue.cc:260] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: NON_VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: true } }
I20240503 02:22:11.519621 25252 ts_tablet_manager.cc:1398] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240503 02:22:11.520491 23339 tablet_copy_service.cc:342] P b0c0b6bc47954db9ada48f0c09ee9959: Request end of tablet copy session dcbb067767bf4060bbac25f4f5fad61a-100904c00d104bcca2956b21edf2c496 received from {username='slave'} at 127.20.218.65:36089
I20240503 02:22:11.520661 23339 tablet_copy_service.cc:433] P b0c0b6bc47954db9ada48f0c09ee9959: ending tablet copy session dcbb067767bf4060bbac25f4f5fad61a-100904c00d104bcca2956b21edf2c496 on tablet 100904c00d104bcca2956b21edf2c496 with peer dcbb067767bf4060bbac25f4f5fad61a
I20240503 02:22:11.667377 23977 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:11.668025 25140 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:11.788792 23448 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:11.789428 25140 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:11.794718 23318 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:11.801937 25292 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:11.826166 23977 raft_consensus.cc:1203] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240503 02:22:12.011842 23186 raft_consensus.cc:1203] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240503 02:22:12.316581 23186 consensus_queue.cc:237] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:12.318522 23318 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:22:12.318648 23977 raft_consensus.cc:1261] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:22:12.319152 25335 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:22:12.319455 25334 consensus_queue.cc:1035] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:22:12.321966 25292 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:12.322656 23977 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:12.324874 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb8aafe8adda4229bbe493e76dc8aed4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:12.325094 23318 raft_consensus.cc:2932] T bb8aafe8adda4229bbe493e76dc8aed4 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:12.326674 23039 catalog_manager.cc:5484] T bb8aafe8adda4229bbe493e76dc8aed4 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 9 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:12.332321 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8aafe8adda4229bbe493e76dc8aed4 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:12.333192 25340 tablet_replica.cc:330] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:12.333659 25340 raft_consensus.cc:2229] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:12.333878 25340 raft_consensus.cc:2258] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:12.334689 25340 ts_tablet_manager.cc:1875] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:12.339207 25340 ts_tablet_manager.cc:1888] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:22:12.339349 25340 log.cc:1196] T bb8aafe8adda4229bbe493e76dc8aed4 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb8aafe8adda4229bbe493e76dc8aed4
I20240503 02:22:12.340133 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet bb8aafe8adda4229bbe493e76dc8aed4 (table rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]) successfully deleted
I20240503 02:22:12.385484 25301 raft_consensus.cc:1050] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: attempting to promote NON_VOTER 8f237b4e800e443fae82db47282e606f to VOTER
I20240503 02:22:12.386191 25301 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:12.388715 23977 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 LEARNER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:12.388715 23318 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:12.389241 23186 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Refusing update from remote peer 7ec52a420ca6438991468d6d68818019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:22:12.389417 25343 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:12.389824 25341 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:12.390213 25301 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:22:12.393628 25341 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.394383 23187 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.397996 23977 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.398141 23318 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.398805 23043 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 5 to 6, 8f237b4e800e443fae82db47282e606f (127.20.218.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ec52a420ca6438991468d6d68818019" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.461273 23448 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29"
dest_uuid: "7ec52a420ca6438991468d6d68818019"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35404
I20240503 02:22:12.461519 23448 raft_consensus.cc:595] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:12.633828 25318 raft_consensus.cc:1050] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959: attempting to promote NON_VOTER dcbb067767bf4060bbac25f4f5fad61a to VOTER
I20240503 02:22:12.634536 25318 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:12.637028 23187 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 LEARNER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:12.637670 23448 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:12.637706 25368 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:12.637869 23977 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:22:12.638314 25371 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:12.638770 25318 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:12.641937 25318 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.642668 23448 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.645522 23977 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.646056 23187 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.646732 23037 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 9 to 10, dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:12.651513 23318 consensus_queue.cc:237] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } }
I20240503 02:22:12.653543 23977 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:22:12.654031 23187 raft_consensus.cc:1261] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Refusing update from remote peer b0c0b6bc47954db9ada48f0c09ee9959: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 02:22:12.654019 25371 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:22:12.654529 25370 consensus_queue.cc:1035] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 02:22:12.656859 25318 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.657677 23187 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.657666 23976 raft_consensus.cc:2932] T 100904c00d104bcca2956b21edf2c496 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.660953 23030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 100904c00d104bcca2956b21edf2c496 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:12.663240 23039 catalog_manager.cc:5484] T 100904c00d104bcca2956b21edf2c496 P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 10 to 11, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } attrs { promote: false } } }
I20240503 02:22:12.666975 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 100904c00d104bcca2956b21edf2c496 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:12.667346 25340 tablet_replica.cc:330] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:12.667520 25340 raft_consensus.cc:2229] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:12.667802 25340 raft_consensus.cc:2258] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:12.668619 25340 ts_tablet_manager.cc:1875] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:12.673144 25340 ts_tablet_manager.cc:1888] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240503 02:22:12.673278 25340 log.cc:1196] T 100904c00d104bcca2956b21edf2c496 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/100904c00d104bcca2956b21edf2c496
I20240503 02:22:12.674005 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 100904c00d104bcca2956b21edf2c496 (table rebalance_test_table_0 [id=ef6a9fac6987401cbf1fc0ba2fe1a01c]) successfully deleted
I20240503 02:22:12.691860 23187 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:12.694015 23976 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:12.694103 23448 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:22:12.694577 25335 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:12.694890 25334 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:12.697749 25335 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.698439 23448 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.699014 23976 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:12.703126 23037 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:12.708464 23187 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:12.711238 23976 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:12.711457 23448 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:22:12.712101 25334 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:12.712543 25292 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:22:12.713616 23120 consensus_peers.cc:485] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a -> Peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): Couldn't send request to peer b0c0b6bc47954db9ada48f0c09ee9959. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:12.715608 23448 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:12.714990 25335 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:12.716212 23976 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:12.719535 23033 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 54227bfcbf9c4ad193075e5eec20a8ad with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:22:12.721724 23037 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 reported cstate change: config changed from index 7 to 8, NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) added. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } } }
I20240503 02:22:12.743860 23448 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29"
dest_uuid: "7ec52a420ca6438991468d6d68818019"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:35414
I20240503 02:22:12.744082 23448 raft_consensus.cc:595] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:22:12.931576 25403 raft_consensus.cc:979] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: : Instructing follower 8f237b4e800e443fae82db47282e606f to start an election
I20240503 02:22:12.931746 25403 raft_consensus.cc:1067] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Signalling peer 8f237b4e800e443fae82db47282e606f to start an election
I20240503 02:22:12.932437 23976 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29"
dest_uuid: "8f237b4e800e443fae82db47282e606f"
 from {username='slave'} at 127.20.218.67:36319
I20240503 02:22:12.932793 23976 raft_consensus.cc:482] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:22:12.932919 23976 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:12.935048 23976 raft_consensus.cc:504] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:12.937172 23187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "8f237b4e800e443fae82db47282e606f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "dcbb067767bf4060bbac25f4f5fad61a"
I20240503 02:22:12.937196 23319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "8f237b4e800e443fae82db47282e606f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b0c0b6bc47954db9ada48f0c09ee9959"
I20240503 02:22:12.937244 23448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb46d2ddb004a6d8498000a326f2d29" candidate_uuid: "8f237b4e800e443fae82db47282e606f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7ec52a420ca6438991468d6d68818019"
I20240503 02:22:12.937431 23319 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:12.937450 23187 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:12.937476 23448 raft_consensus.cc:3032] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:22:12.937559 23448 raft_consensus.cc:729] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ec52a420ca6438991468d6d68818019, State: Running, Role: LEADER
I20240503 02:22:12.937880 23448 consensus_queue.cc:260] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:12.938313 23448 raft_consensus.cc:3037] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:22:12.939532 23187 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f237b4e800e443fae82db47282e606f in term 2.
I20240503 02:22:12.940304 23319 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f237b4e800e443fae82db47282e606f in term 2.
I20240503 02:22:12.936731 23976 leader_election.cc:288] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [CANDIDATE]: Term 2 election: Requested vote from peers dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687), b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401), 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461)
I20240503 02:22:12.940857 23910 leader_election.cc:302] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8f237b4e800e443fae82db47282e606f, b0c0b6bc47954db9ada48f0c09ee9959, dcbb067767bf4060bbac25f4f5fad61a; no voters: 
I20240503 02:22:12.941066 23448 raft_consensus.cc:2454] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f237b4e800e443fae82db47282e606f in term 2.
I20240503 02:22:12.941256 25111 raft_consensus.cc:2781] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:22:12.941761 25111 raft_consensus.cc:686] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 2 LEADER]: Becoming Leader. State: Replica: 8f237b4e800e443fae82db47282e606f, State: Running, Role: LEADER
I20240503 02:22:12.942320 25111 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:12.945672 23039 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f reported cstate change: term changed from 1 to 2, leader changed from 7ec52a420ca6438991468d6d68818019 (127.20.218.67) to 8f237b4e800e443fae82db47282e606f (127.20.218.68). New cstate: current_term: 2 leader_uuid: "8f237b4e800e443fae82db47282e606f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:22:13.323336 25426 ts_tablet_manager.cc:902] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Initiating tablet copy from peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687)
I20240503 02:22:13.324599 25426 tablet_copy_client.cc:308] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Beginning tablet copy session from remote peer at address 127.20.218.65:35687
I20240503 02:22:13.325424 23207 tablet_copy_service.cc:140] P dcbb067767bf4060bbac25f4f5fad61a: Received BeginTabletCopySession request for tablet 54227bfcbf9c4ad193075e5eec20a8ad from peer b0c0b6bc47954db9ada48f0c09ee9959 ({username='slave'} at 127.20.218.66:55997)
I20240503 02:22:13.325611 23207 tablet_copy_service.cc:161] P dcbb067767bf4060bbac25f4f5fad61a: Beginning new tablet copy session on tablet 54227bfcbf9c4ad193075e5eec20a8ad from peer b0c0b6bc47954db9ada48f0c09ee9959 at {username='slave'} at 127.20.218.66:55997: session id = b0c0b6bc47954db9ada48f0c09ee9959-54227bfcbf9c4ad193075e5eec20a8ad
I20240503 02:22:13.327854 23207 tablet_copy_source_session.cc:205] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:22:13.328881 25426 ts_tablet_manager.cc:1875] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 02:22:13.333272 25426 ts_tablet_manager.cc:1888] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 02:22:13.333561 25426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54227bfcbf9c4ad193075e5eec20a8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:22:13.335706 25426 tablet_copy_client.cc:783] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 0 data blocks...
I20240503 02:22:13.336033 25426 tablet_copy_client.cc:647] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Starting download of 1 WAL segments...
I20240503 02:22:13.337777 25426 tablet_copy_client.cc:523] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:22:13.340152 25426 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap starting.
I20240503 02:22:13.380996 25426 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:22:13.381441 25426 tablet_bootstrap.cc:492] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Bootstrap complete.
I20240503 02:22:13.381685 25426 ts_tablet_manager.cc:1367] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Time spent bootstrapping tablet: real 0.042s	user 0.040s	sys 0.003s
I20240503 02:22:13.382308 25426 raft_consensus.cc:348] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:13.382521 25426 raft_consensus.cc:729] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b0c0b6bc47954db9ada48f0c09ee9959, State: Initialized, Role: LEARNER
I20240503 02:22:13.382683 25426 consensus_queue.cc:260] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: NON_VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: true } }
I20240503 02:22:13.383176 25426 ts_tablet_manager.cc:1398] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240503 02:22:13.384043 23207 tablet_copy_service.cc:342] P dcbb067767bf4060bbac25f4f5fad61a: Request end of tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-54227bfcbf9c4ad193075e5eec20a8ad received from {username='slave'} at 127.20.218.66:55997
I20240503 02:22:13.384210 23207 tablet_copy_service.cc:433] P dcbb067767bf4060bbac25f4f5fad61a: ending tablet copy session b0c0b6bc47954db9ada48f0c09ee9959-54227bfcbf9c4ad193075e5eec20a8ad on tablet 54227bfcbf9c4ad193075e5eec20a8ad with peer b0c0b6bc47954db9ada48f0c09ee9959
I20240503 02:22:13.407078 23187 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Refusing update from remote peer 8f237b4e800e443fae82db47282e606f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:22:13.407747 25111 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:13.473024 23448 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Refusing update from remote peer 8f237b4e800e443fae82db47282e606f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:22:13.473743 25111 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:13.478777 23319 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Refusing update from remote peer 8f237b4e800e443fae82db47282e606f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:22:13.483974 25450 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:22:13.716919 23319 raft_consensus.cc:1203] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240503 02:22:14.097950 23977 consensus_queue.cc:237] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } }
I20240503 02:22:14.099828 23319 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Refusing update from remote peer 8f237b4e800e443fae82db47282e606f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:22:14.099838 23187 raft_consensus.cc:1261] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Refusing update from remote peer 8f237b4e800e443fae82db47282e606f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:22:14.100427 25487 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:14.100749 25486 consensus_queue.cc:1035] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:22:14.103248 25445 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:14.103825 23319 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:14.103945 23187 raft_consensus.cc:2932] T beb46d2ddb004a6d8498000a326f2d29 P dcbb067767bf4060bbac25f4f5fad61a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:14.108242 23043 catalog_manager.cc:5484] T beb46d2ddb004a6d8498000a326f2d29 P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 6 to 8, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 2 leader_uuid: "8f237b4e800e443fae82db47282e606f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } }
I20240503 02:22:14.110026 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet beb46d2ddb004a6d8498000a326f2d29 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:14.113920 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet beb46d2ddb004a6d8498000a326f2d29 with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:14.114557 25493 tablet_replica.cc:330] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:14.114754 25493 raft_consensus.cc:2229] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:14.114959 25493 raft_consensus.cc:2258] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:14.115978 25493 ts_tablet_manager.cc:1875] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:14.120548 25493 ts_tablet_manager.cc:1888] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:22:14.120671 25493 log.cc:1196] T beb46d2ddb004a6d8498000a326f2d29 P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/beb46d2ddb004a6d8498000a326f2d29
I20240503 02:22:14.121438 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet beb46d2ddb004a6d8498000a326f2d29 (table rebalance_test_table_1 [id=cb78199de04549c190735df06ccf23f2]) successfully deleted
I20240503 02:22:14.180369 25465 raft_consensus.cc:1050] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a: attempting to promote NON_VOTER b0c0b6bc47954db9ada48f0c09ee9959 to VOTER
I20240503 02:22:14.181143 25465 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:14.183658 23319 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 LEARNER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:14.183653 23977 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:14.183681 23448 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:22:14.184214 25495 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:14.184584 25465 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:14.184890 25464 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:22:14.190954 25464 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.191788 23319 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.192426 23977 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.196475 23037 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 reported cstate change: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.196735 23448 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "7ec52a420ca6438991468d6d68818019" member_type: VOTER last_known_addr { host: "127.20.218.67" port: 41461 } attrs { replace: true } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.204028 23187 consensus_queue.cc:237] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } }
I20240503 02:22:14.206015 23977 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:14.206600 23319 raft_consensus.cc:1261] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Refusing update from remote peer dcbb067767bf4060bbac25f4f5fad61a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240503 02:22:14.206616 25495 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:14.207185 25465 consensus_queue.cc:1035] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:22:14.209681 25464 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P dcbb067767bf4060bbac25f4f5fad61a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.210420 23977 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.213701 23319 raft_consensus.cc:2932] T 54227bfcbf9c4ad193075e5eec20a8ad P b0c0b6bc47954db9ada48f0c09ee9959 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.214341 23033 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54227bfcbf9c4ad193075e5eec20a8ad with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:22:14.215834 23039 catalog_manager.cc:5484] T 54227bfcbf9c4ad193075e5eec20a8ad P 8f237b4e800e443fae82db47282e606f reported cstate change: config changed from index 9 to 10, VOTER 7ec52a420ca6438991468d6d68818019 (127.20.218.67) evicted. New cstate: current_term: 1 leader_uuid: "dcbb067767bf4060bbac25f4f5fad61a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dcbb067767bf4060bbac25f4f5fad61a" member_type: VOTER last_known_addr { host: "127.20.218.65" port: 35687 } } peers { permanent_uuid: "8f237b4e800e443fae82db47282e606f" member_type: VOTER last_known_addr { host: "127.20.218.68" port: 41297 } attrs { promote: false } } peers { permanent_uuid: "b0c0b6bc47954db9ada48f0c09ee9959" member_type: VOTER last_known_addr { host: "127.20.218.66" port: 34401 } attrs { promote: false } } }
I20240503 02:22:14.220362 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet 54227bfcbf9c4ad193075e5eec20a8ad with delete_type TABLET_DATA_TOMBSTONED (TS 7ec52a420ca6438991468d6d68818019 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59830
I20240503 02:22:14.220757 25493 tablet_replica.cc:330] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: stopping tablet replica
I20240503 02:22:14.220929 25493 raft_consensus.cc:2229] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:22:14.221169 25493 raft_consensus.cc:2258] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:22:14.221959 25493 ts_tablet_manager.cc:1875] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:22:14.226377 25493 ts_tablet_manager.cc:1888] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240503 02:22:14.226490 25493 log.cc:1196] T 54227bfcbf9c4ad193075e5eec20a8ad P 7ec52a420ca6438991468d6d68818019: Deleting WAL directory at /tmp/dist-test-taskEtZc7o/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714702283380610-21353-0/raft_consensus-itest-cluster/ts-2/wal/wals/54227bfcbf9c4ad193075e5eec20a8ad
I20240503 02:22:14.227196 23033 catalog_manager.cc:4830] TS 7ec52a420ca6438991468d6d68818019 (127.20.218.67:41461): tablet 54227bfcbf9c4ad193075e5eec20a8ad (table rebalance_test_table_2 [id=ecc97614d35442b19110122c005991ec]) successfully deleted
I20240503 02:22:14.605077 23957 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:22:14.610903 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:14.615628 23429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:14.618806 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:14.777748 23429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:14.784685 23957 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:22:14.794498 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:14.797398 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:14.948762 23429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:14.953012 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:14.960706 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:14.963021 23957 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:22:15.131886 23429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:15.138366 23957 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:22:15.139390 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:15.142848 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:15.287770 23429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:22:15.290715 23957 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:22:15.301362 23167 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 02:22:15.319229 23297 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:22:15.514205 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23105
I20240503 02:22:15.525460 23226 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:22:15.697238 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23105
W20240503 02:22:15.728608 23911 connection.cc:535] server connection from 127.20.218.65:55307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:22:15.728672 23913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.218.65:35687 (error 108)
W20240503 02:22:15.729059 23255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.218.65:35687 (error 108)
I20240503 02:22:15.729413 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23236
W20240503 02:22:15.729797 23253 connection.cc:535] server connection from 127.20.218.65:36089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:22:15.732192 23255 consensus_peers.cc:485] T 100904c00d104bcca2956b21edf2c496 P b0c0b6bc47954db9ada48f0c09ee9959 -> Peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): Couldn't send request to peer dcbb067767bf4060bbac25f4f5fad61a. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:35687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:22:15.732529 23913 consensus_peers.cc:485] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f -> Peer dcbb067767bf4060bbac25f4f5fad61a (127.20.218.65:35687): Couldn't send request to peer dcbb067767bf4060bbac25f4f5fad61a. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.65:35687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:15.736268 23359 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:22:15.901408 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23236
W20240503 02:22:15.931768 23913 connection.cc:535] server connection from 127.20.218.66:40425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:22:15.932220 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23368
I20240503 02:22:15.938380 23491 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 02:22:15.988463 23910 consensus_peers.cc:485] T beb46d2ddb004a6d8498000a326f2d29 P 8f237b4e800e443fae82db47282e606f -> Peer b0c0b6bc47954db9ada48f0c09ee9959 (127.20.218.66:34401): Couldn't send request to peer b0c0b6bc47954db9ada48f0c09ee9959. Status: Network error: Client connection negotiation failed: client connection to 127.20.218.66:34401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:22:16.093822 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23368
I20240503 02:22:16.122635 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23895
I20240503 02:22:16.128728 24016 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:22:16.284057 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23895
W20240503 02:22:16.311071 23032 connection.cc:535] server connection from 127.20.218.68:55319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:22:16.311344 21353 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu pid 23015
I20240503 02:22:16.312469 23076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:22:16.475697 21353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEtZc7o/build/asan/bin/kudu with pid 23015
2024-05-03T02:22:16Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (27699 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (50453 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (653130 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>

 1 FAILED TEST